[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 233d9dcf4e5 Updating config from bot 233d9dcf4e5 is described below commit 233d9dcf4e536dc385427670475558d4b013ca8d Author: github-actions AuthorDate: Fri Mar 17 08:44:22 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25876.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25876.json b/scripts/ci/pr-bot/state/pr-state/pr-25876.json new file mode 100644 index 000..9c2aa5aa212 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-25876.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": true, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 0ef09ee9a90 Updating config from bot 0ef09ee9a90 is described below commit 0ef09ee9a900815a47feb7d5087fddedec987063 Author: github-actions AuthorDate: Fri Mar 17 09:33:57 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25861.json | 10 ++ 1 file changed, 10 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25861.json b/scripts/ci/pr-bot/state/pr-state/pr-25861.json new file mode 100644 index 000..6272b4e797d --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-25861.json @@ -0,0 +1,10 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": { +"python": "AnandInguva" + }, + "nextAction": "Reviewers", + "stopReviewerNotifications": false, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new f1d7551db93 Updating config from bot f1d7551db93 is described below commit f1d7551db93933c43e13a9ee81178d08315dcc74 Author: github-actions AuthorDate: Fri Mar 17 09:33:59 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 16b8d6aece3..205b47517ce 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -1,7 +1,7 @@ { "label": "python", "dateOfLastReviewAssignment": { -"AnandInguva": 1678959242992, +"AnandInguva": 1679045635592, "yeandy": 1665802753763, "TheNeuralBit": 1667896849319, "ryanthompson591": 1670002443548,
[beam-starter-typescript] branch dependabot/npm_and_yarn/typescript-5.0.2 created (now 37937e1)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/typescript-5.0.2 in repository https://gitbox.apache.org/repos/asf/beam-starter-typescript.git at 37937e1 Bump typescript from 4.7.3 to 5.0.2 No new revisions were added by this update.
[beam] branch master updated (213b3ed6bf1 -> cbc1a7661bd)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 213b3ed6bf1 Use self hosted hadoop docker image in hdfsIntegrationTest (#25867) add cbc1a7661bd Revert "Update sqlalchemy requirement in /sdks/python (#25855)" (#25872) No new revisions were added by this update. Summary of changes: sdks/python/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch dependabot/pip/sdks/python/pytest-xdist-gte-2.5.0-and-lt-4 updated (9f1c216ead8 -> 72474f63988)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/pip/sdks/python/pytest-xdist-gte-2.5.0-and-lt-4 in repository https://gitbox.apache.org/repos/asf/beam.git discard 9f1c216ead8 Update pytest-xdist requirement in /sdks/python add eac926878c2 Use autoSharding from config when BigQueryIO.Write is created. (#25862) add 738825c90fb Revert "Adding support for DLQ for ZetaSQL (#25426)" add bb5e5d0 Merge pull request #25863: Revert "Adding support for DLQ for ZetaSQL (#25426)" add 213b3ed6bf1 Use self hosted hadoop docker image in hdfsIntegrationTest (#25867) add cbc1a7661bd Revert "Update sqlalchemy requirement in /sdks/python (#25855)" (#25872) add 72474f63988 Update pytest-xdist requirement in /sdks/python This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (9f1c216ead8) \ N -- N -- N refs/heads/dependabot/pip/sdks/python/pytest-xdist-gte-2.5.0-and-lt-4 (72474f63988) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/coders/RowCoderGenerator.java | 12 +-- .../extensions/sql/zetasql/BeamZetaSqlCalcRel.java | 112 + .../sql/zetasql/BeamZetaSqlCalcRelTest.java| 70 - .../sql/zetasql/ZetaSqlDialectSpecTest.java| 14 +-- .../io/gcp/bigquery/BigQuerySchemaIOProvider.java | 9 +- ...ueryStorageWriteApiSchemaTransformProvider.java | 23 +++-- .../io/hdfs_integration_test/docker-compose.yml| 4 +- sdks/python/setup.py | 2 +- 8 files changed, 58 insertions(+), 188 deletions(-)
[beam] branch master updated: Minor fix: move the getattr out of the loop as it is not needed to be inside (#25859)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new b2afbe0b26f Minor fix: move the getattr out of the loop as it is not needed to be inside (#25859) b2afbe0b26f is described below commit b2afbe0b26f7ea27d9073e582f913dd912b6d82c Author: Amir Fayazi AuthorDate: Fri Mar 17 05:00:36 2023 -0700 Minor fix: move the getattr out of the loop as it is not needed to be inside (#25859) --- sdks/python/apache_beam/ml/inference/pytorch_inference.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/ml/inference/pytorch_inference.py b/sdks/python/apache_beam/ml/inference/pytorch_inference.py index 818dd8325dc..cc6f3ea6be7 100644 --- a/sdks/python/apache_beam/ml/inference/pytorch_inference.py +++ b/sdks/python/apache_beam/ml/inference/pytorch_inference.py @@ -380,7 +380,7 @@ def make_keyed_tensor_model_fn(model_fn: str) -> KeyedTensorInferenceFn: batched_tensors = torch.stack(key_to_tensor_list[key]) batched_tensors = _convert_to_device(batched_tensors, device) key_to_batched_tensors[key] = batched_tensors -pred_fn = getattr(model, model_fn) + pred_fn = getattr(model, model_fn) predictions = pred_fn(**key_to_batched_tensors, **inference_args) return utils._convert_to_result(batch, predictions, model_id)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new fb3b8f3da03 Updating config from bot fb3b8f3da03 is described below commit fb3b8f3da033650289dafc708b22e62325448a27 Author: github-actions AuthorDate: Fri Mar 17 12:13:55 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25632.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25632.json b/scripts/ci/pr-bot/state/pr-state/pr-25632.json index a7c19975bb1..e5769456548 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-25632.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-25632.json @@ -1,8 +1,8 @@ { "commentedAboutFailingChecks": false, "reviewersAssignedForLabels": { -"java": "robertwb", -"io": "Abacn" +"java": "kennknowles", +"io": "johnjcasey" }, "nextAction": "Reviewers", "stopReviewerNotifications": false,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 2af1a985f9c Updating config from bot 2af1a985f9c is described below commit 2af1a985f9c7d22fb8c600ac4f342d5c067af5ad Author: github-actions AuthorDate: Fri Mar 17 12:14:00 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index 0b5db1fb436..1a73d7e75cf 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -2,7 +2,7 @@ "label": "io", "dateOfLastReviewAssignment": { "chamikaramj": 1678989904221, -"johnjcasey": 1678822446183, +"johnjcasey": 1679055233734, "pabloem": 1678831493228, "Abacn": 1678977301745, "ahmedabu98": 1678995247594
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new ea329f9f39c Updating config from bot ea329f9f39c is described below commit ea329f9f39cee2ca4d71479a6d10a91b029fd830 Author: github-actions AuthorDate: Fri Mar 17 12:13:58 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index e21e2c7f4f8..162a5162dd0 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -2,7 +2,7 @@ "label": "java", "dateOfLastReviewAssignment": { "lukecwik": 1679014126796, -"kennknowles": 1678989904221, +"kennknowles": 1679055233118, "robertwb": 1679006042236, "kileys": 1674428678843, "apilloud": 1678822446183
[beam] branch master updated (b2afbe0b26f -> 821a77beb00)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from b2afbe0b26f Minor fix: move the getattr out of the loop as it is not needed to be inside (#25859) add 821a77beb00 Multi language/runinference example (#25083) No new revisions were added by this update. Summary of changes: .../inference/multi_language_inference/README.md | 69 +++ .../beam/examples/MultiLangRunInference.java | 92 ++ .../multi_language_custom_transform}/__init__.py | 3 +- .../composite_transform.py | 122 +++ .../multi_language_custom_transform/setup.py} | 12 +- .../documentation/ml/multi-language-inference.md | 133 + .../site/content/en/documentation/ml/overview.md | 5 +- .../partials/section-menu/en/documentation.html| 3 +- 8 files changed, 428 insertions(+), 11 deletions(-) create mode 100644 sdks/python/apache_beam/examples/inference/multi_language_inference/README.md create mode 100644 sdks/python/apache_beam/examples/inference/multi_language_inference/last_word_prediction/src/main/java/org/apache/beam/examples/MultiLangRunInference.java copy {.test-infra/jenkins/dependency_check => sdks/python/apache_beam/examples/inference/multi_language_inference/multi_language_custom_transform/multi_language_custom_transform}/__init__.py (99%) create mode 100644 sdks/python/apache_beam/examples/inference/multi_language_inference/multi_language_custom_transform/multi_language_custom_transform/composite_transform.py copy sdks/python/apache_beam/{io/gcp/pubsublite/__init__.py => examples/inference/multi_language_inference/multi_language_custom_transform/setup.py} (84%) create mode 100644 website/www/site/content/en/documentation/ml/multi-language-inference.md
[beam] branch master updated (821a77beb00 -> d9f81b8592d)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 821a77beb00 Multi language/runinference example (#25083) add d9f81b8592d Upgrade tox from 3.x.x to 4.x.x (#25785) No new revisions were added by this update. Summary of changes: .github/workflows/python_tests.yml | 6 +++--- release/src/main/scripts/build_release_candidate.sh | 3 +-- sdks/python/MANIFEST.in | 1 + sdks/python/gen_protos.py | 13 - sdks/python/tox.ini | 18 +- 5 files changed, 18 insertions(+), 23 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 7f43e53bb9d Updating config from bot 7f43e53bb9d is described below commit 7f43e53bb9ddbb77d2ed2f31a1a7c257045767f4 Author: github-actions AuthorDate: Fri Mar 17 14:05:13 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 205b47517ce..3a3ccd76bde 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -8,7 +8,7 @@ "tvalentyn": 1678995263753, "pabloem": 1678995247594, "y1chi": 1667002607045, -"damccorm": 1678979106036, +"damccorm": 1679061908460, "jrmccluskey": 1679014132643 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 1d38dfe3530 Updating config from bot 1d38dfe3530 is described below commit 1d38dfe3530ae9ad45ebcb23820993327e1bc005 Author: github-actions AuthorDate: Fri Mar 17 14:05:10 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25833.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25833.json b/scripts/ci/pr-bot/state/pr-state/pr-25833.json index 6272b4e797d..24d0e10a64c 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-25833.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-25833.json @@ -1,10 +1,10 @@ { "commentedAboutFailingChecks": false, "reviewersAssignedForLabels": { -"python": "AnandInguva" +"python": "damccorm" }, "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 18137020fcc Updating config from bot 18137020fcc is described below commit 18137020fccee6e7d22e2f1a3d28b5873f00209b Author: github-actions AuthorDate: Fri Mar 17 14:12:18 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25877.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25877.json b/scripts/ci/pr-bot/state/pr-state/pr-25877.json new file mode 100644 index 000..9c2aa5aa212 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-25877.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": true, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
[beam] branch master updated (d9f81b8592d -> 167a4c5fb59)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from d9f81b8592d Upgrade tox from 3.x.x to 4.x.x (#25785) add 167a4c5fb59 Remove changes from MANIFEST.in (#25877) No new revisions were added by this update. Summary of changes: sdks/python/MANIFEST.in | 1 - 1 file changed, 1 deletion(-)
[beam] branch master updated (167a4c5fb59 -> b4ee5bd60e0)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 167a4c5fb59 Remove changes from MANIFEST.in (#25877) add b4ee5bd60e0 Update pytest-xdist requirement in /sdks/python (#25833) No new revisions were added by this update. Summary of changes: sdks/python/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 62c96d63600 Updating config from bot 62c96d63600 is described below commit 62c96d636001f0c23febe375d88d0e7d1dc0150f Author: github-actions AuthorDate: Fri Mar 17 17:28:55 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25886.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25886.json b/scripts/ci/pr-bot/state/pr-state/pr-25886.json new file mode 100644 index 000..9c2aa5aa212 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-25886.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": true, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new a3959b919ae Updating config from bot a3959b919ae is described below commit a3959b919aebc1c73377d9c2c3952c04582beb63 Author: github-actions AuthorDate: Fri Mar 17 18:05:10 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25885.json | 10 ++ 1 file changed, 10 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25885.json b/scripts/ci/pr-bot/state/pr-state/pr-25885.json new file mode 100644 index 000..d26f23226c7 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-25885.json @@ -0,0 +1,10 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": { +"java": "robertwb" + }, + "nextAction": "Reviewers", + "stopReviewerNotifications": false, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 71c86fde341 Updating config from bot 71c86fde341 is described below commit 71c86fde341dc5af98ed063cb0c49b3dd590a79e Author: github-actions AuthorDate: Fri Mar 17 18:05:13 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index 162a5162dd0..66c8e626c2a 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -3,7 +3,7 @@ "dateOfLastReviewAssignment": { "lukecwik": 1679014126796, "kennknowles": 1679055233118, -"robertwb": 1679006042236, +"robertwb": 1679076308290, "kileys": 1674428678843, "apilloud": 1678822446183 }
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 09d4e9c7913 Updating config from bot 09d4e9c7913 is described below commit 09d4e9c79138d348042272fac1bfc1c6b7bc347b Author: github-actions AuthorDate: Fri Mar 17 18:35:44 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25216.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25216.json b/scripts/ci/pr-bot/state/pr-state/pr-25216.json new file mode 100644 index 000..242a48d7d3b --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-25216.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": true, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": false, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
[beam] branch master updated: Onboard Performance tests using the Change point analysis tool. (#25754)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 6f82f34eeb1 Onboard Performance tests using the Change point analysis tool. (#25754) 6f82f34eeb1 is described below commit 6f82f34eeb14097810f3e8faee10221ac1c54aa3 Author: Anand Inguva <34158215+ananding...@users.noreply.github.com> AuthorDate: Fri Mar 17 14:49:07 2023 -0400 Onboard Performance tests using the Change point analysis tool. (#25754) --- .github/workflows/run_perf_alert_tool.yml | 24 + .../python/apache_beam/testing/analyzers/README.md | 9 ++-- .../apache_beam/testing/analyzers/constants.py | 2 +- .../testing/analyzers/github_issues_utils.py | 3 +- .../apache_beam/testing/analyzers/perf_analysis.py | 17 --- .../testing/analyzers/perf_analysis_utils.py | 20 .../testing/analyzers/tests_config.yaml| 59 ++ .../benchmarks/cloudml/cloudml_benchmark_test.py | 24 - 8 files changed, 93 insertions(+), 65 deletions(-) diff --git a/.github/workflows/run_perf_alert_tool.yml b/.github/workflows/run_perf_alert_tool.yml index 30779e765f1..65e89ca5f5f 100644 --- a/.github/workflows/run_perf_alert_tool.yml +++ b/.github/workflows/run_perf_alert_tool.yml @@ -17,12 +17,12 @@ # To learn more about GitHub Actions in Apache Beam check the CI.md -name: Run performance alerting tool on Python load/performance/benchmark tests. +name: Performance alerting tool on Python load/performance/benchmark tests. on: + workflow_dispatch: schedule: - cron: '5 22 * * *' - jobs: python_run_change_point_analysis: name: Run Change Point Analysis. @@ -34,6 +34,11 @@ jobs: uses: actions/setup-python@v4 with: python-version: 3.8 + - name: Authenticate on GCP +uses: google-github-actions/setup-gcloud@v0 +with: + service_account_key: ${{ secrets.GCP_SA_KEY }} + export_default_credentials: true - name: Get Apache Beam Build dependencies working-directory: ./sdks/python run: pip install pip setuptools --upgrade && pip install -r build-requirements.txt @@ -43,17 +48,16 @@ jobs: - name: Install signal-processing-algorithms run: pip install signal-processing-algorithms - name: Install pandas, yaml, requests -run: pip install pandas PyYAML requests -# - name: Run Change Point Analysis. -#working-directory: ./sdks/python/apache_beam/testing/analyzers -#shell: bash -#run: python analysis.py -#env: -# GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} +run: pip install pandas db-dtypes PyYAML requests + - name: Run Change Point Analysis. +working-directory: ./sdks/python/apache_beam/testing/analyzers +shell: bash +run: python perf_analysis.py +env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Run change point analysis tests. working-directory: ./sdks/python/apache_beam/testing/analyzers shell: bash run: pytest perf_analysis_test.py env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - diff --git a/sdks/python/apache_beam/testing/analyzers/README.md b/sdks/python/apache_beam/testing/analyzers/README.md index 9f50d9797b5..71351fe3e57 100644 --- a/sdks/python/apache_beam/testing/analyzers/README.md +++ b/sdks/python/apache_beam/testing/analyzers/README.md @@ -41,7 +41,8 @@ please follow the below structure. ``` # the test_1 must be a unique id. test_1: - test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks + test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152 + test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks source: big_query metrics_dataset: beam_run_inference metrics_table: torch_inference_imagenet_results_resnet152 @@ -53,9 +54,9 @@ test_1: num_runs_in_change_point_window: 30 # optional parameter ``` -**NOTE**: `test_name` should be in the format `apache_beam.foo.bar`. It should point to a single test target. +**NOTE**: `test_target` is optional. It is used for identifying the test that was causing the regression. -**Note**: If the source is **BigQuery**, the metrics_dataset, metrics_table, project and metric_name should match with the values defined for performance/load tests. +**Note**: If the source is **BigQuery**, the `metrics_dataset`, `metrics_table`, `project` and `metric_name` should match with the values defined for performance/load tests. The above example uses this [test configuration](https://github.com/apache/beam/blob/0a91d139dea4276dc46176c4cdcdfce210fc50c4/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L30) to fill u
[beam] 01/01: Merge pull request #25871: Update default splitting behavior to get fewer, larger, splits
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit de578e606e5d07eb9efe3ffc993ed0d9d11e Merge: 6f82f34eeb1 a9826ee2db1 Author: Kenneth Knowles AuthorDate: Fri Mar 17 12:10:18 2023 -0700 Merge pull request #25871: Update default splitting behavior to get fewer, larger, splits .../core/src/main/java/org/apache/beam/sdk/io/Read.java| 14 -- 1 file changed, 8 insertions(+), 6 deletions(-)
[beam] branch master updated (6f82f34eeb1 -> de578e606e5)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 6f82f34eeb1 Onboard Performance tests using the Change point analysis tool. (#25754) add a9826ee2db1 Update default splitting behavior to get fewer, larger, splits new de578e606e5 Merge pull request #25871: Update default splitting behavior to get fewer, larger, splits The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../core/src/main/java/org/apache/beam/sdk/io/Read.java| 14 -- 1 file changed, 8 insertions(+), 6 deletions(-)
[beam] branch master updated: Make conversion to Iterable tighter (#25866)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 7555d989799 Make conversion to Iterable tighter (#25866) 7555d989799 is described below commit 7555d9897996362a82e6e5d25202ddf29931338d Author: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com> AuthorDate: Fri Mar 17 15:23:55 2023 -0400 Make conversion to Iterable tighter (#25866) --- sdks/python/apache_beam/typehints/native_type_compatibility.py| 2 +- .../apache_beam/typehints/native_type_compatibility_test.py | 8 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/typehints/native_type_compatibility.py b/sdks/python/apache_beam/typehints/native_type_compatibility.py index f0ef1b2e2c3..d03d5db4045 100644 --- a/sdks/python/apache_beam/typehints/native_type_compatibility.py +++ b/sdks/python/apache_beam/typehints/native_type_compatibility.py @@ -201,7 +201,7 @@ def convert_collections_to_typing(typ): typ = typing.Iterator[typ.__args__] elif hasattr(typ, 'send') and hasattr(typ, 'throw'): typ = typing.Generator[typ.__args__] -else: +elif _match_is_exactly_iterable(typ): typ = typing.Iterable[typ.__args__] return typ diff --git a/sdks/python/apache_beam/typehints/native_type_compatibility_test.py b/sdks/python/apache_beam/typehints/native_type_compatibility_test.py index 013fa354b82..89045ae7a25 100644 --- a/sdks/python/apache_beam/typehints/native_type_compatibility_test.py +++ b/sdks/python/apache_beam/typehints/native_type_compatibility_test.py @@ -156,6 +156,14 @@ class NativeTypeCompatibilityTest(unittest.TestCase): 'nested iterable', tuple[bytes, collections.abc.Iterable[int]], typehints.Tuple[bytes, typehints.Iterable[int]]), + ( + 'iterable over tuple', + collections.abc.Iterable[tuple[str, int]], + typehints.Iterable[typehints.Tuple[str, int]]), + ( + 'mapping not caught', + collections.abc.Mapping[str, int], + collections.abc.Mapping[str, int]), ] for test_case in test_cases:
[beam] branch master updated: Add support for CPython 3.11 opcodes (#25795)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new fad7f96dc85 Add support for CPython 3.11 opcodes (#25795) fad7f96dc85 is described below commit fad7f96dc85047eb1a01536f817565d6f959ee35 Author: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com> AuthorDate: Fri Mar 17 16:14:44 2023 -0400 Add support for CPython 3.11 opcodes (#25795) * Add support for CPython 3.11 opcodes * Remove unused utility function * Uncomment test * Drop camel case form * Add addition documentation comments --- sdks/python/apache_beam/typehints/opcodes.py | 36 +- .../apache_beam/typehints/trivial_inference.py | 129 +++-- 2 files changed, 151 insertions(+), 14 deletions(-) diff --git a/sdks/python/apache_beam/typehints/opcodes.py b/sdks/python/apache_beam/typehints/opcodes.py index 1ac3538c691..5a35b56b932 100644 --- a/sdks/python/apache_beam/typehints/opcodes.py +++ b/sdks/python/apache_beam/typehints/opcodes.py @@ -178,7 +178,9 @@ binary_lshift = inplace_lshift = binary_rshift = inplace_rshift = pop_top binary_and = inplace_and = symmetric_binary_op binary_xor = inplace_xor = symmetric_binary_op -binary_or = inpalce_or = symmetric_binary_op +binary_or = inplace_or = symmetric_binary_op + +binary_op = symmetric_binary_op def store_subscr(unused_state, unused_args): @@ -403,6 +405,8 @@ import_from = push_value(Any) def load_global(state, arg): + if (sys.version_info.major, sys.version_info.minor) >= (3, 11): +arg = arg >> 1 state.stack.append(state.get_global(arg)) @@ -428,10 +432,18 @@ def gen_start(state, arg): def load_closure(state, arg): + # The arg is no longer offset by len(covar_names) as of 3.11 + # See https://docs.python.org/3/library/dis.html#opcode-LOAD_CLOSURE + if (sys.version_info.major, sys.version_info.minor) >= (3, 11): +arg -= len(state.co.co_varnames) state.stack.append(state.get_closure(arg)) def load_deref(state, arg): + # The arg is no longer offset by len(covar_names) as of 3.11 + # See https://docs.python.org/3/library/dis.html#opcode-LOAD_DEREF + if (sys.version_info.major, sys.version_info.minor) >= (3, 11): +arg -= len(state.co.co_varnames) state.stack.append(state.closure_type(arg)) @@ -440,9 +452,19 @@ def make_function(state, arg): """ # TODO(luke-zhu): Handle default argument types globals = state.f.__globals__ # Inherits globals from the current frame - func_name = state.stack[-1].value - func_code = state.stack[-2].value - pop_count = 2 + tos = state.stack[-1].value + # In Python 3.11 lambdas no longer have fully qualified names on the stack, + # so we check for this case (AKA the code is top of stack.) + if isinstance(tos, types.CodeType): +func_name = None +func_code = tos +pop_count = 1 +is_lambda = True + else: +func_name = tos +func_code = state.stack[-2].value +pop_count = 2 +is_lambda = False closure = None # arg contains flags, with corresponding stack values if positive. # https://docs.python.org/3.6/library/dis.html#opcode-MAKE_FUNCTION @@ -450,8 +472,12 @@ def make_function(state, arg): if arg & 0x08: # Convert types in Tuple constraint to a tuple of CPython cells. # https://stackoverflow.com/a/44670295 +if is_lambda: + closureTuplePos = -2 +else: + closureTuplePos = -3 closure = tuple((lambda _: lambda: _)(t).__closure__[0] -for t in state.stack[-3].tuple_types) +for t in state.stack[closureTuplePos].tuple_types) func = types.FunctionType(func_code, globals, name=func_name, closure=closure) diff --git a/sdks/python/apache_beam/typehints/trivial_inference.py b/sdks/python/apache_beam/typehints/trivial_inference.py index f69a87192e3..f4b350e8f05 100644 --- a/sdks/python/apache_beam/typehints/trivial_inference.py +++ b/sdks/python/apache_beam/typehints/trivial_inference.py @@ -126,11 +126,12 @@ class Const(object): class FrameState(object): """Stores the state of the frame at a particular point of execution. """ - def __init__(self, f, local_vars=None, stack=()): + def __init__(self, f, local_vars=None, stack=(), kw_names=None): self.f = f self.co = f.__code__ self.vars = list(local_vars) self.stack = list(stack) +self.kw_names = kw_names def __eq__(self, other): return isinstance(other, FrameState) and self.__dict__ == other.__dict__ @@ -139,7 +140,7 @@ class FrameState(object): return hash(tuple(sorted(self.__dict__.items( def copy(self): -return FrameState(self.f, self.vars, self.stack) +return FrameState(self.f, self.vars, self.stack, self.kw_names) def const_type(self, i): return Const(self.co.co_consts[i]) @@ -35
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 4353db95c7a Updating config from bot 4353db95c7a is described below commit 4353db95c7aa286d38d502c28f457d63060661d8 Author: github-actions AuthorDate: Fri Mar 17 21:37:34 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25873.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25873.json b/scripts/ci/pr-bot/state/pr-state/pr-25873.json index 499fd94a3a0..ffa9ca4fec0 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-25873.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-25873.json @@ -4,7 +4,7 @@ "java": "lukecwik" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": false } \ No newline at end of file
[beam] branch master updated: Adding DLQ support for ZetaSQL (#25873)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new e8fad9ca2ba Adding DLQ support for ZetaSQL (#25873) e8fad9ca2ba is described below commit e8fad9ca2ba80220b4817ddba95aabc0a48067dd Author: Pablo Estrada AuthorDate: Fri Mar 17 18:14:03 2023 -0400 Adding DLQ support for ZetaSQL (#25873) --- .../extensions/sql/zetasql/BeamZetaSqlCalcRel.java | 126 - 1 file changed, 97 insertions(+), 29 deletions(-) diff --git a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRel.java b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRel.java index 744fbd0bcd4..dee87d370e6 100644 --- a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRel.java +++ b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamZetaSqlCalcRel.java @@ -37,18 +37,24 @@ import org.apache.beam.sdk.annotations.Internal; import org.apache.beam.sdk.extensions.sql.impl.BeamSqlPipelineOptions; import org.apache.beam.sdk.extensions.sql.impl.QueryPlanner.QueryParameters; import org.apache.beam.sdk.extensions.sql.impl.rel.AbstractBeamCalcRel; +import org.apache.beam.sdk.extensions.sql.impl.rel.BeamSqlRelUtils; import org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils; import org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BeamBigQuerySqlDialect; import org.apache.beam.sdk.extensions.sql.meta.provider.bigquery.BeamSqlUnparseContext; import org.apache.beam.sdk.schemas.FieldAccessDescriptor; import org.apache.beam.sdk.schemas.Schema; +import org.apache.beam.sdk.schemas.utils.SelectHelpers; import org.apache.beam.sdk.transforms.DoFn; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.transforms.windowing.BoundedWindow; import org.apache.beam.sdk.values.PCollection; import org.apache.beam.sdk.values.PCollectionList; +import org.apache.beam.sdk.values.PCollectionTuple; +import org.apache.beam.sdk.values.POutput; import org.apache.beam.sdk.values.Row; +import org.apache.beam.sdk.values.TupleTag; +import org.apache.beam.sdk.values.TupleTagList; import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptCluster; import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelTraitSet; import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.RelNode; @@ -64,7 +70,6 @@ import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.fun.SqlStdO import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.parser.SqlParserPos; import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions; import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList; -import org.checkerframework.checker.nullness.qual.NonNull; import org.checkerframework.checker.nullness.qual.Nullable; import org.joda.time.Duration; import org.joda.time.Instant; @@ -83,6 +88,9 @@ public class BeamZetaSqlCalcRel extends AbstractBeamCalcRel { private static final int MAX_PENDING_WINDOW = 32; private final BeamSqlUnparseContext context; + private static final TupleTag rows = new TupleTag("output") {}; + private static final TupleTag errors = new TupleTag("errors") {}; + private static String columnName(int i) { return "_" + i; } @@ -101,21 +109,36 @@ public class BeamZetaSqlCalcRel extends AbstractBeamCalcRel { @Override public PTransform, PCollection> buildPTransform() { -return new Transform(); +return buildPTransform(null); + } + + @Override + public PTransform, PCollection> buildPTransform( + @Nullable PTransform, ? extends POutput> errorsTransformer) { +return new Transform(errorsTransformer); } @AutoValue abstract static class TimestampedFuture { -private static TimestampedFuture create(Instant t, Future f) { - return new AutoValue_BeamZetaSqlCalcRel_TimestampedFuture(t, f); +private static TimestampedFuture create(Instant t, Future f, Row r) { + return new AutoValue_BeamZetaSqlCalcRel_TimestampedFuture(t, f, r); } abstract Instant timestamp(); abstract Future future(); + +abstract Row row(); } private class Transform extends PTransform, PCollection> { + +private final @Nullable PTransform, ? extends POutput> errorsTransformer; + +Transform(@Nullable PTransform, ? extends POutput> errorsTransformer) { + this.errorsTransformer = errorsTransformer; +} + @Override public PCollection expand(PCollectionList pinput) { Preconditions.checkArgument( @@ -135,9 +158,10 @@ public class BeamZetaSqlCalcRel extends AbstractBeamC
[beam] branch master updated: Optimize change stream connector with more efficient batching and blind writes, and add transaction/query tags (#25718)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new ac57b8709d3 Optimize change stream connector with more efficient batching and blind writes, and add transaction/query tags (#25718) ac57b8709d3 is described below commit ac57b8709d3f2db597a5d9e6339545b0f2036a3a Author: ChangyuLi28 <35211213+changyul...@users.noreply.github.com> AuthorDate: Fri Mar 17 15:13:27 2023 -0700 Optimize change stream connector with more efficient batching and blind writes, and add transaction/query tags (#25718) * Optimize change stream connector with more efficient batching and blind writes, and add transaction/query tags * Optimize change stream connector with more efficient batching and blind writes, and add transaction/query tags * Apply commit deadline only for metadata table writes - Co-authored-by: Changyu Li --- .../MetadataSpannerConfigFactory.java | 2 +- .../action/ChildPartitionsRecordAction.java| 3 +- .../action/DetectNewPartitionsAction.java | 19 -- .../changestreams/dao/PartitionMetadataDao.java| 68 +++--- .../action/ChildPartitionsRecordActionTest.java| 4 +- .../dao/PartitionMetadataDaoTest.java | 29 +++-- 6 files changed, 75 insertions(+), 50 deletions(-) diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/MetadataSpannerConfigFactory.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/MetadataSpannerConfigFactory.java index 83965b1bfaa..56c67f3194f 100644 --- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/MetadataSpannerConfigFactory.java +++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/MetadataSpannerConfigFactory.java @@ -84,7 +84,7 @@ public class MetadataSpannerConfigFactory { ValueProvider commitDeadline = primaryConfig.getCommitDeadline(); if (commitDeadline != null) { - config = config.withCommitDeadline(StaticValueProvider.of(commitDeadline.get())); + config = config.withCommitDeadline(StaticValueProvider.of(Duration.standardSeconds(60))); } ValueProvider maxCumulativeBackoff = primaryConfig.getMaxCumulativeBackoff(); diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/ChildPartitionsRecordAction.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/ChildPartitionsRecordAction.java index 70286b41778..7fb69d0e7ab 100644 --- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/ChildPartitionsRecordAction.java +++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/ChildPartitionsRecordAction.java @@ -155,7 +155,8 @@ public class ChildPartitionsRecordAction { } else { return false; } -}) +}, +"InsertChildPartition") .getResult(); if (insertedRow && isSplit) { metrics.incPartitionRecordSplitCount(); diff --git a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/DetectNewPartitionsAction.java b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/DetectNewPartitionsAction.java index 934210250f5..73967d2a2a7 100644 --- a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/DetectNewPartitionsAction.java +++ b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/action/DetectNewPartitionsAction.java @@ -146,15 +146,22 @@ public class DetectNewPartitionsAction { OutputReceiver receiver, Timestamp minWatermark, TreeMap> batches) { +List batchPartitionsDifferentCreatedAt = new ArrayList<>(); +int numTimestampsHandledSofar = 0; for (Map.Entry> batch : batches.entrySet()) { + numTimestampsHandledSofar++; final Timestamp batchCreatedAt = batch.getKey(); - final List batchPartitions = batch.getValue(); - - final Timestamp scheduledAt = updateBatchToScheduled(batchPartitions); - if (!tracker.tryClaim(batchCreatedAt)) { -return ProcessContinuation.stop(); + final List batchPartitionsSameCreatedAt = batch.getValue(); + batchPartitionsDifferentCreatedAt.addAll(batchPartitionsSameCreatedAt); + if (batchPartitionsDifferentCreatedAt.size() >= 200 + || numTimestampsH
[beam] branch asf-site updated: Publishing website 2023/03/17 22:17:23 at commit e8fad9c
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new ad9fd1e0c8b Publishing website 2023/03/17 22:17:23 at commit e8fad9c ad9fd1e0c8b is described below commit ad9fd1e0c8bc7bed7e486204818d33fb6625a1bb Author: jenkins AuthorDate: Fri Mar 17 22:17:24 2023 + Publishing website 2023/03/17 22:17:23 at commit e8fad9c --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 5ee9202fbf7..77029a1c729 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9"; xmlns:xhtml="http://www.w3.org/1999/xhtml";>/blog/beam-2.46.0/2023-03-17T10:40:29-04:00/categories/blog/2023-03-17T10:40:29-04:00/blog/2023-03-17T10:40:29-04:00/categories/2023-03-17T10:40:29-04:00/catego [...] \ No newline at end of file +http://www.sitemaps.org/schemas/sitemap/0.9"; xmlns:xhtml="http://www.w3.org/1999/xhtml";>/blog/beam-2.46.0/2023-03-17T18:14:03-04:00/categories/blog/2023-03-17T18:14:03-04:00/blog/2023-03-17T18:14:03-04:00/categories/2023-03-17T18:14:03-04:00/catego [...] \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 0c3fb3fe36a Updating config from bot 0c3fb3fe36a is described below commit 0c3fb3fe36ac07a155f7e53fb0673144d6118ec8 Author: github-actions AuthorDate: Fri Mar 17 23:37:23 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25891.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25891.json b/scripts/ci/pr-bot/state/pr-state/pr-25891.json new file mode 100644 index 000..9c2aa5aa212 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-25891.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": true, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new ef37b6a9726 Updating config from bot ef37b6a9726 is described below commit ef37b6a97268c5c95a39f18bca8d35c948e5ddbf Author: github-actions AuthorDate: Sat Mar 18 00:49:09 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25890.json | 10 ++ 1 file changed, 10 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25890.json b/scripts/ci/pr-bot/state/pr-state/pr-25890.json new file mode 100644 index 000..62db1666e3b --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-25890.json @@ -0,0 +1,10 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": { +"python": "pabloem" + }, + "nextAction": "Reviewers", + "stopReviewerNotifications": false, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new d84775ff581 Updating config from bot d84775ff581 is described below commit d84775ff581e70885207a1e0345da8202e0fa595 Author: github-actions AuthorDate: Sat Mar 18 00:49:11 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 3a3ccd76bde..8287f4956ab 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -6,7 +6,7 @@ "TheNeuralBit": 1667896849319, "ryanthompson591": 1670002443548, "tvalentyn": 1678995263753, -"pabloem": 1678995247594, +"pabloem": 1679100548105, "y1chi": 1667002607045, "damccorm": 1679061908460, "jrmccluskey": 1679014132643
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new a1ceda206e8 Updating config from bot a1ceda206e8 is described below commit a1ceda206e8354edd5762dc289b993583f693cf3 Author: github-actions AuthorDate: Sat Mar 18 01:08:46 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-25893.json | 8 1 file changed, 8 insertions(+) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-25893.json b/scripts/ci/pr-bot/state/pr-state/pr-25893.json new file mode 100644 index 000..9c2aa5aa212 --- /dev/null +++ b/scripts/ci/pr-bot/state/pr-state/pr-25893.json @@ -0,0 +1,8 @@ +{ + "commentedAboutFailingChecks": false, + "reviewersAssignedForLabels": {}, + "nextAction": "Author", + "stopReviewerNotifications": true, + "remindAfterTestsPass": [], + "committerAssigned": false +} \ No newline at end of file
[beam] branch master updated: Disable firestore integration tests temporarily (#25891)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 4e96e700c2d Disable firestore integration tests temporarily (#25891) 4e96e700c2d is described below commit 4e96e700c2d08865ee05b6ac19de76232a8d78ad Author: Kenneth Knowles AuthorDate: Fri Mar 17 20:37:27 2023 -0700 Disable firestore integration tests temporarily (#25891) --- sdks/java/io/google-cloud-platform/build.gradle | 5 + 1 file changed, 5 insertions(+) diff --git a/sdks/java/io/google-cloud-platform/build.gradle b/sdks/java/io/google-cloud-platform/build.gradle index 8b264a9c8b3..fa696306f9e 100644 --- a/sdks/java/io/google-cloud-platform/build.gradle +++ b/sdks/java/io/google-cloud-platform/build.gradle @@ -203,6 +203,11 @@ task integrationTest(type: Test, dependsOn: processTestResources) { exclude '**/BigQueryIOStorageWriteIT.class' exclude '**/BigQueryToTableIT.class' exclude '**/BigQueryIOJsonTest.class' + + // Failing due to Firestore service-side changes + // https://github.com/apache/beam/issues/25851 + exclude '**/firestore/it/**/*.class' + maxParallelForks 4 classpath = sourceSets.test.runtimeClasspath testClassesDirs = sourceSets.test.output.classesDirs
[beam] branch asf-site updated: Publishing website 2023/03/18 04:15:45 at commit 4e96e70
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 2a365356c84 Publishing website 2023/03/18 04:15:45 at commit 4e96e70 2a365356c84 is described below commit 2a365356c84dad2de072c86126415cae9c6c2092 Author: jenkins AuthorDate: Sat Mar 18 04:15:45 2023 + Publishing website 2023/03/18 04:15:45 at commit 4e96e70 --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 77029a1c729..8b74f7dc6e6 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9"; xmlns:xhtml="http://www.w3.org/1999/xhtml";>/blog/beam-2.46.0/2023-03-17T18:14:03-04:00/categories/blog/2023-03-17T18:14:03-04:00/blog/2023-03-17T18:14:03-04:00/categories/2023-03-17T18:14:03-04:00/catego [...] \ No newline at end of file +http://www.sitemaps.org/schemas/sitemap/0.9"; xmlns:xhtml="http://www.w3.org/1999/xhtml";>/blog/beam-2.46.0/2023-03-17T20:37:27-07:00/categories/blog/2023-03-17T20:37:27-07:00/blog/2023-03-17T20:37:27-07:00/categories/2023-03-17T20:37:27-07:00/catego [...] \ No newline at end of file