[beam] branch master updated (4f1dcfb224d -> cf904dc72aa)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 4f1dcfb224d Dask runner tests action (#24324) add 5802c77d383 Add a portable runner that renders pipelines as a dot graph. add c1d82dbfff2 Add basic tests for render runner. add c01d63096b7 Add the ability to pass a pipeline proto directly. add f179587039f lint add 4c2c78def15 Fix typo. add 63e566ffc79 Serve the graph when output file is not specified. add 2002d9c3927 Serve the graph when output file is not specified. add 3db0ac9ba36 Fix parsing of standalone protos. add ad3e5242f7b Support reading from GCS. add 7285242cb83 Add text logging. add 6749fc01647 fix typo. add e3ab5e75ae5 Some lint and yapf. add 5c8026671f2 Fix dot detection logic. add 38e0e3b3397 fix error detected by lint add 20f60443f46 Make gcs an optional dependency. add 42ea58fb276 return rather than sys.exit add a88784e1442 lint add d57812a3860 Apply suggestions from code review add e625c317bfb More cleanup, mypy. add de117c5940c lint add cf904dc72aa Merge pull request #24037 Add a portable runner that renders pipelines as a dot graph. No new revisions were added by this update. Summary of changes: CHANGES.md | 2 + sdks/python/apache_beam/pipeline.py| 3 +- .../portability/fn_api_runner/translations.py | 16 +- sdks/python/apache_beam/runners/render.py | 537 + sdks/python/apache_beam/runners/render_test.py | 89 sdks/python/scripts/run_pylint.sh | 1 + 6 files changed, 642 insertions(+), 6 deletions(-) create mode 100644 sdks/python/apache_beam/runners/render.py create mode 100644 sdks/python/apache_beam/runners/render_test.py
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 669fa0367c2 Updating config from bot
669fa0367c2 is described below
commit 669fa0367c229f1187858c2265c32526a767c443
Author: github-actions
AuthorDate: Wed Nov 23 04:56:15 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24326.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24326.json
b/scripts/ci/pr-bot/state/pr-state/pr-24326.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24326.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: Dask runner tests action (#24324)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 4f1dcfb224d Dask runner tests action (#24324)
4f1dcfb224d is described below
commit 4f1dcfb224dc398e6d29530b1bb811bbc688c1d3
Author: Pablo Estrada
AuthorDate: Tue Nov 22 19:44:15 2022 -0800
Dask runner tests action (#24324)
---
.github/workflows/dask_runner_tests.yml | 104
1 file changed, 104 insertions(+)
diff --git a/.github/workflows/dask_runner_tests.yml
b/.github/workflows/dask_runner_tests.yml
new file mode 100644
index 000..382437c0c9a
--- /dev/null
+++ b/.github/workflows/dask_runner_tests.yml
@@ -0,0 +1,104 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Dask Runner Tests
+
+on:
+ schedule:
+- cron: 'H H * * *'
+ pull_request:
+branches: ['master', 'release-*']
+tags: 'v*'
+paths: ['sdks/python/apache_beam/runners/dask/**']
+
+# This allows a subsequently queued workflow run to interrupt previous runs
+concurrency:
+ group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label ||
github.head_ref || github.ref }}'
+ cancel-in-progress: true
+
+jobs:
+
+ build_python_sdk_source:
+name: 'Build python source distribution'
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout code
+uses: actions/checkout@v3
+ - name: Install python
+uses: actions/setup-python@v4
+with:
+ python-version: 3.7
+ - name: Get build dependencies
+working-directory: ./sdks/python
+run: pip install pip setuptools --upgrade && pip install -r
build-requirements.txt
+ - name: Build source
+working-directory: ./sdks/python
+run: python setup.py sdist
+ - name: Rename source file
+working-directory: ./sdks/python/dist
+run: mv $(ls | grep "apache-beam.*tar\.gz") apache-beam-source.tar.gz
+ - name: Upload compressed sources as artifacts
+uses: actions/upload-artifact@v3
+with:
+ name: python_sdk_source
+ path: sdks/python/dist/apache-beam-source.tar.gz
+
+ python_unit_tests:
+name: 'Python Unit Tests'
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+os: [ubuntu-latest, macos-latest, windows-latest]
+params: [
+ {"py_ver": "3.7", "tox_env": "py37"},
+ {"py_ver": "3.8", "tox_env": "py38"},
+ {"py_ver": "3.9", "tox_env": "py39"},
+ {"py_ver": "3.10", "tox_env": "py310" },
+]
+steps:
+ - name: Checkout code
+uses: actions/checkout@v3
+ - name: Install python
+uses: actions/setup-python@v4
+with:
+ python-version: ${{ matrix.params.py_ver }}
+ - name: Get build dependencies
+working-directory: ./sdks/python
+run: pip install -r build-requirements.txt
+ - name: Install tox
+run: pip install tox
+ - name: Install SDK with dask
+working-directory: ./sdks/python
+run: pip install setuptools --upgrade && pip install -e
.[gcp,dask,test]
+ - name: Run tests basic unix
+if: startsWith(matrix.os, 'ubuntu') || startsWith(matrix.os, 'macos')
+working-directory: ./sdks/python
+run: tox -c tox.ini -e ${{ matrix.params.tox_env }}-dask
+ - name: Run tests basic windows
+if: startsWith(matrix.os, 'windows')
+working-directory: ./sdks/python
+run: tox -c tox.ini -e ${{ matrix.params.tox_env }}-win-dask
+ - name: Upload test logs
+uses: actions/upload-artifact@v3
+if: always()
+with:
+ name: pytest-${{matrix.os}}-${{matrix.params.py_ver}}
+ path: sdks/python/pytest**.xml
+
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1265244a3bf Updating config from bot
1265244a3bf is described below
commit 1265244a3bfbeffe8b4c808afdcce477caf2d55b
Author: github-actions
AuthorDate: Wed Nov 23 02:34:07 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24317.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24317.json
b/scripts/ci/pr-bot/state/pr-state/pr-24317.json
new file mode 100644
index 000..1867c087f49
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24317.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "tvalentyn",
+"io": "ahmedabu98"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 477494dffc8 Updating config from bot 477494dffc8 is described below commit 477494dffc8d3aa7d30ddc64a3718f9d58562ccb Author: github-actions AuthorDate: Wed Nov 23 02:34:09 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 53a2f054c3f..c5a68ae0bcd 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -5,7 +5,7 @@ "yeandy": 1665802753763, "TheNeuralBit": 1667896849319, "ryanthompson591": 1669076246392, -"tvalentyn": 1669070164376, +"tvalentyn": 1669170845621, "pabloem": 1669071896928, "y1chi": 1667002607045 }
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new d76d254ae2e Updating config from bot d76d254ae2e is described below commit d76d254ae2ef6700017113b534b402af6020bdf0 Author: github-actions AuthorDate: Wed Nov 23 02:34:12 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index 3f9eb6114f0..ba95749b35b 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -5,6 +5,6 @@ "johnjcasey": 1669151092253, "pabloem": 1669050393201, "Abacn": 1669057497006, -"ahmedabu98": 1668897324082 +"ahmedabu98": 1669170845621 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new bdae933cbc0 Updating config from bot
bdae933cbc0 is described below
commit bdae933cbc07b5aeb0d8bec9beed9e98d4e5e260
Author: github-actions
AuthorDate: Wed Nov 23 01:32:54 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index 256292c401d..ecea3922b17 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1669107898121,
+"damccorm": 1669167170903,
"Abacn": 1669162573554
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new d17ab846694 Updating config from bot
d17ab846694 is described below
commit d17ab84669433262f4f899798df2c0fa435fba82
Author: github-actions
AuthorDate: Wed Nov 23 01:32:52 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24324.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24324.json
b/scripts/ci/pr-bot/state/pr-state/pr-24324.json
new file mode 100644
index 000..8a92900947e
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24324.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (9bad75f1fee -> 730b391f29c)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 9bad75f1fee [Playground] Remove example bucket (#24198)
add 730b391f29c Extract Go and Python Beam symbols for Playground (#23378)
No new revisions were added by this update.
Summary of changes:
.gitignore | 7 +-
build.gradle.kts |11 +
.../frontend/lib/assets/assets.gen.dart| 132 +
.../lib/components/login/login_content.dart| 2 +-
.../frontend/lib/components/profile/avatar.dart| 2 +-
.../lib/components/profile/profile_content.dart| 2 +-
.../frontend/lib/models/unit_content.g.dart|16 +
.../lib/pages/tour/widgets/group_title.dart| 2 +-
.../frontend/lib/pages/tour/widgets/unit.dart | 2 +-
.../frontend/lib/pages/welcome/screen.dart | 2 +-
.../models/get_content_tree_response.g.dart|16 +
.../repositories/models/get_sdks_response.g.dart |14 +
.../frontend/lib/repositories/models/group.g.dart |15 +
.../frontend/lib/repositories/models/module.g.dart |24 +
.../frontend/lib/repositories/models/node.g.dart |23 +
.../frontend/lib/repositories/models/unit.g.dart |13 +
learning/tour-of-beam/frontend/pubspec.yaml| 2 +-
playground/frontend/Dockerfile |18 +-
playground/frontend/README.md | 140 +-
playground/frontend/build.gradle |59 +-
.../config.example.dart} |25 +-
playground/frontend/lib/main.dart | 2 +
.../playground_components/assets/symbols/go.g.yaml | 6553 ++
.../assets/symbols/python.g.yaml | 12128 +++
.../playground_components/build.gradle.kts |97 +-
.../lib/playground_components.dart | 5 +-
.../lib/src/assets/assets.gen.dart | 145 +
.../lib/src/controllers/playground_controller.dart |16 +
.../controllers/snippet_editing_controller.dart|30 +-
.../playground_components.dart => locator.dart}|12 +-
.../lib/src/models/sdk.g.dart |12 +
.../symbols_dictionary.dart} |14 +-
.../lib/src/notifications/notification.dart| 2 +-
.../src/{constants => }/playground_components.dart | 6 +
.../symbols/loaders/abstract.dart} |12 +-
.../symbols/loaders/map.dart} |30 +-
.../lib/src/services/symbols/loaders/yaml.dart |69 +
.../lib/src/services/symbols/symbols_notifier.dart |53 +
.../lib/src/widgets/drag_handle.dart | 4 +-
.../lib/src/widgets/logo.dart | 4 +-
.../lib/src/widgets/reset_button.dart | 4 +-
.../lib/src/widgets/toggle_theme_button.dart | 4 +-
.../lib/src/widgets/toggle_theme_icon_button.dart |28 +-
.../frontend/playground_components/pubspec.yaml| 8 +-
.../src/common/example_repository_mock.mocks.dart | 142 +
.../http_example_loader_test.mocks.dart| 284 +
.../controllers/playground_controller_test.dart| 9 +-
.../playground_controller_test.mocks.dart | 357 +
.../repositories/code_repository_test.mocks.dart | 139 +
.../example_repository_test.mocks.dart | 165 +
.../src/services/symbols/loaders/yaml_test.dart|63 +
.../services/symbols/symbols_notifier_test.dart|59 +
.../playground_components/test/tools/common.dart |90 +
.../extract_symbols_go_test.dart} |24 +-
.../test/tools/extract_symbols_go/go.golden.yaml |17 +
.../extract_symbols_go/sdk_mock/directory/file2.go |19 +
.../sdk_mock/directory/ignore.txt |18 +
.../tools/extract_symbols_go/sdk_mock/file1.go |73 +
.../extract_symbols_python_test.dart} |23 +-
.../extract_symbols_python/python.golden.yaml | 9 +
.../sdk_mock/directory/file2.py|17 +
.../tools/extract_symbols_python/sdk_mock/file1.py |47 +
.../tools/extract_symbols_go/extract_symbols_go.go | 257 +
.../tools/extract_symbols_go/go.mod|20 +
.../tools/extract_symbols_go/go.sum| 3 +
.../extract_symbols_python.py | 124 +
playground/frontend/pubspec.lock |11 +-
playground/frontend/pubspec.yaml | 1 +
.../states/example_selector_state_test.mocks.dart |67 +
69 files changed, 21657 insertions(+), 146 deletions(-)
create mode 100644 learning/tour-of-beam/frontend/lib/assets/assets.gen.dart
create mode 100644
learning/tour-of-beam/frontend/lib/models/unit_content.g.dart
create mode 100644
[beam] branch master updated (c49571c61e4 -> 9bad75f1fee)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from c49571c61e4 Merge pull request #24231: Fixed display of the selected example add 9bad75f1fee [Playground] Remove example bucket (#24198) No new revisions were added by this update. Summary of changes: playground/terraform/README.md | 1 - .../terraform/infrastructure/buckets/main.tf | 32 -- .../terraform/infrastructure/buckets/output.tf | 31 -- .../terraform/infrastructure/buckets/variables.tf | 38 -- playground/terraform/infrastructure/main.tf| 14 +--- playground/terraform/infrastructure/variables.tf | 17 -- playground/terraform/main.tf | 4 --- playground/terraform/output.tf | 25 -- playground/terraform/variables.tf | 18 -- 9 files changed, 1 insertion(+), 179 deletions(-) delete mode 100644 playground/terraform/infrastructure/buckets/main.tf delete mode 100644 playground/terraform/infrastructure/buckets/output.tf delete mode 100644 playground/terraform/infrastructure/buckets/variables.tf
[beam] branch master updated (d93ece1d0ed -> c49571c61e4)
This is an automated email from the ASF dual-hosted git repository. apilloud pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from d93ece1d0ed Merge pull request #23958: [Website] update share button and ADD_CASE_STUDY.md add 2c3973311eb pg_23865 fix selected example at list add e61cd6a57c9 Expand all categories that contain a selected example (#23865) add c49571c61e4 Merge pull request #24231: Fixed display of the selected example No new revisions were added by this update. Summary of changes: .../examples/components/example_list/category_expansion_panel.dart| 4 ++-- .../examples/components/example_list/expansion_panel_item.dart| 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated (0c6d147409d -> d93ece1d0ed)
This is an automated email from the ASF dual-hosted git repository. apilloud pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 0c6d147409d Merge pull request #24315: fixed typo add f8c57d6cc1a [Website] change share-your-story link, add text to ADD_CASE_STUDY.md, add video to some case-studies add d93ece1d0ed Merge pull request #23958: [Website] update share button and ADD_CASE_STUDY.md No new revisions were added by this update. Summary of changes: website/ADD_CASE_STUDY.md | 1 + website/www/site/content/en/case-studies/intuit.md | 5 + website/www/site/content/en/case-studies/ricardo.md | 5 + website/www/site/layouts/case-studies/list.html | 2 +- website/www/site/layouts/partials/quotes/quote.html | 2 +- 5 files changed, 13 insertions(+), 2 deletions(-)
[beam] branch master updated (71c6e314d27 -> 0c6d147409d)
This is an automated email from the ASF dual-hosted git repository. apilloud pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 71c6e314d27 Add rootCaCertificate option to SplunkIO (#24229) add 7ddf55a545c fixed typo new 0c6d147409d Merge pull request #24315: fixed typo The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/sdk/io/gcp/bigquery/BigQueryResourceNaming.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #24315: fixed typo
This is an automated email from the ASF dual-hosted git repository. apilloud pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 0c6d147409d2dc2e20e82f9668b47bc5d54ded11 Merge: 71c6e314d27 7ddf55a545c Author: Andrew Pilloud AuthorDate: Tue Nov 22 16:29:54 2022 -0800 Merge pull request #24315: fixed typo .../org/apache/beam/sdk/io/gcp/bigquery/BigQueryResourceNaming.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 14430fa2aab Updating config from bot
14430fa2aab is described below
commit 14430fa2aab2e62ce7a51ea763083ae39b6d6313
Author: github-actions
AuthorDate: Wed Nov 23 00:27:28 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-23651.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-23651.json
b/scripts/ci/pr-bot/state/pr-state/pr-23651.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-23651.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (0a91d139dea -> 71c6e314d27)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 0a91d139dea Golang SpannerIO Implementation (#23285) add 71c6e314d27 Add rootCaCertificate option to SplunkIO (#24229) No new revisions were added by this update. Summary of changes: .../beam/sdk/io/splunk/CustomX509TrustManager.java | 98 + .../beam/sdk/io/splunk/HttpEventPublisher.java | 53 +++-- .../beam/sdk/io/splunk/SplunkEventWriter.java | 46 +++- .../org/apache/beam/sdk/io/splunk/SplunkIO.java| 65 --- .../sdk/io/splunk/CustomX509TrustManagerTest.java | 80 ++ .../beam/sdk/io/splunk/HttpEventPublisherTest.java | 119 +++-- .../test/resources/SplunkTestCerts/PrivateKey.pem | 28 + .../SplunkTestCerts/RecognizedCertificate.crt.txt | 26 + .../test/resources/SplunkTestCerts/RootCA.crt.txt | 31 ++ .../resources/SplunkTestCerts/RootCA_2.crt.txt | 13 +++ .../SplunkTestCerts/RootCA_PrivateKey.pem | 52 + .../UnrecognizedCertificate.crt.txt| 21 12 files changed, 600 insertions(+), 32 deletions(-) create mode 100644 sdks/java/io/splunk/src/main/java/org/apache/beam/sdk/io/splunk/CustomX509TrustManager.java create mode 100644 sdks/java/io/splunk/src/test/java/org/apache/beam/sdk/io/splunk/CustomX509TrustManagerTest.java create mode 100644 sdks/java/io/splunk/src/test/resources/SplunkTestCerts/PrivateKey.pem create mode 100644 sdks/java/io/splunk/src/test/resources/SplunkTestCerts/RecognizedCertificate.crt.txt create mode 100644 sdks/java/io/splunk/src/test/resources/SplunkTestCerts/RootCA.crt.txt create mode 100644 sdks/java/io/splunk/src/test/resources/SplunkTestCerts/RootCA_2.crt.txt create mode 100644 sdks/java/io/splunk/src/test/resources/SplunkTestCerts/RootCA_PrivateKey.pem create mode 100644 sdks/java/io/splunk/src/test/resources/SplunkTestCerts/UnrecognizedCertificate.crt.txt
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 08a8d8a1e01 Updating config from bot
08a8d8a1e01 is described below
commit 08a8d8a1e019a8b458794abc6f2336f626c77f5f
Author: github-actions
AuthorDate: Wed Nov 23 00:25:32 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-23357.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-23357.json
b/scripts/ci/pr-bot/state/pr-state/pr-23357.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-23357.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 7b6debc6ea2 Updating config from bot 7b6debc6ea2 is described below commit 7b6debc6ea262b72ca2ca6f0ed96c9768d09d2f4 Author: github-actions AuthorDate: Wed Nov 23 00:16:17 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index 251d749cc3a..79249cef6fd 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -4,7 +4,7 @@ "lukecwik": 1669145648881, "kennknowles": 1669145635270, "robertwb": 1669062868143, -"kileys": 1669057497006, +"kileys": 1669162573554, "apilloud": 1669151092253 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new fafa9985c21 Updating config from bot
fafa9985c21 is described below
commit fafa9985c212e268f0693933dcb07519f42e7c25
Author: github-actions
AuthorDate: Wed Nov 23 00:16:19 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index a47920d7f4b..256292c401d 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -2,6 +2,6 @@
"label": "build",
"dateOfLastReviewAssignment": {
"damccorm": 1669107898121,
-"Abacn": 1669062868143
+"Abacn": 1669162573554
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 4af06d9ec97 Updating config from bot
4af06d9ec97 is described below
commit 4af06d9ec97ba3f15d00b4dd87fa05b69d18f0b2
Author: github-actions
AuthorDate: Wed Nov 23 00:16:15 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24320.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24320.json
b/scripts/ci/pr-bot/state/pr-state/pr-24320.json
new file mode 100644
index 000..0baec2a00e6
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24320.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "kileys",
+"build": "Abacn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new f1880e459e5 Updating config from bot
f1880e459e5 is described below
commit f1880e459e569f402a08dc1de6013015eca60100
Author: github-actions
AuthorDate: Tue Nov 22 23:44:07 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24321.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24321.json
b/scripts/ci/pr-bot/state/pr-state/pr-24321.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24321.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 13b9d503ddf Updating config from bot 13b9d503ddf is described below commit 13b9d503ddf614024ca0d3e273f7c95453c66371 Author: github-actions AuthorDate: Tue Nov 22 23:34:41 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json b/scripts/ci/pr-bot/state/reviewers-for-label-go.json index b34a114e6b6..4a4c3cbe373 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json @@ -4,7 +4,7 @@ "lostluck": 1669160070573, "jrmccluskey": 1669008889648, "youngoli": 1657688896155, -"damccorm": 1669007109184, +"damccorm": 1669160077395, "riteshghorse": 1669133112874 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 13496e3417a Updating config from bot
13496e3417a is described below
commit 13496e3417a18c88eb58e7d06b2b1827dd76f290
Author: github-actions
AuthorDate: Tue Nov 22 23:34:39 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24318.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24318.json
b/scripts/ci/pr-bot/state/pr-state/pr-24318.json
new file mode 100644
index 000..2deda2c6f85
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24318.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2622a5ee40d Updating config from bot
2622a5ee40d is described below
commit 2622a5ee40d9d0813af568ec2c6c633c32f207b6
Author: github-actions
AuthorDate: Tue Nov 22 23:34:32 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24319.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24319.json
b/scripts/ci/pr-bot/state/pr-state/pr-24319.json
new file mode 100644
index 000..8ea94810368
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24319.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "lostluck"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 56d58308480 Updating config from bot
56d58308480 is described below
commit 56d5830848067e55b807ff667f9ebcc09debcb28
Author: github-actions
AuthorDate: Tue Nov 22 23:34:35 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
index 8e01b316bd4..b34a114e6b6 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json
@@ -1,7 +1,7 @@
{
"label": "go",
"dateOfLastReviewAssignment": {
-"lostluck": 1669007102820,
+"lostluck": 1669160070573,
"jrmccluskey": 1669008889648,
"youngoli": 1657688896155,
"damccorm": 1669007109184,
[beam] branch dependabot/go_modules/sdks/github.com/aws/aws-sdk-go-v2/config-1.18.3 created (now b7f6c89c285)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/aws/aws-sdk-go-v2/config-1.18.3 in repository https://gitbox.apache.org/repos/asf/beam.git at b7f6c89c285 Bump github.com/aws/aws-sdk-go-v2/config from 1.18.2 to 1.18.3 in /sdks No new revisions were added by this update.
[beam] branch dependabot/go_modules/sdks/github.com/aws/aws-sdk-go-v2/credentials-1.13.3 created (now f631dffb6a6)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/aws/aws-sdk-go-v2/credentials-1.13.3 in repository https://gitbox.apache.org/repos/asf/beam.git at f631dffb6a6 Bump github.com/aws/aws-sdk-go-v2/credentials in /sdks No new revisions were added by this update.
[beam] branch master updated: Golang SpannerIO Implementation (#23285)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 0a91d139dea Golang SpannerIO Implementation (#23285) 0a91d139dea is described below commit 0a91d139dea4276dc46176c4cdcdfce210fc50c4 Author: Chris Gavin AuthorDate: Wed Nov 23 09:56:40 2022 +1100 Golang SpannerIO Implementation (#23285) --- sdks/go.mod | 4 +- sdks/go.sum | 6 + sdks/go/pkg/beam/io/spannerio/spanner.go | 241 + sdks/go/pkg/beam/io/spannerio/spanner_test.go | 249 ++ 4 files changed, 499 insertions(+), 1 deletion(-) diff --git a/sdks/go.mod b/sdks/go.mod index 061250fc0dd..b9d0728c711 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -61,6 +61,8 @@ require ( gopkg.in/yaml.v2 v2.4.0 ) +require cloud.google.com/go/spanner v1.36.0 + require ( cloud.google.com/go/bigtable v1.18.0 github.com/tetratelabs/wazero v1.0.0-pre.3 @@ -92,7 +94,7 @@ require ( github.com/aws/aws-sdk-go-v2/service/ssooidc v1.13.8 // indirect github.com/aws/aws-sdk-go-v2/service/sts v1.17.4 // indirect github.com/cenkalti/backoff/v4 v4.1.3 // indirect - github.com/census-instrumentation/opencensus-proto v0.2.1 // indirect + github.com/census-instrumentation/opencensus-proto v0.3.0 // indirect github.com/cespare/xxhash/v2 v2.1.2 // indirect github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4 // indirect github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1 // indirect diff --git a/sdks/go.sum b/sdks/go.sum index acee0826903..2c1d96beab1 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -68,6 +68,8 @@ cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIA cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU= cloud.google.com/go/pubsub v1.26.0 h1:Y/HcMxVXgkUV2pYeLMUkclMg0ue6U0jVyI5xEARQ4zA= cloud.google.com/go/pubsub v1.26.0/go.mod h1:QgBH3U/jdJy/ftjPhTkyXNj543Tin1pRYcdcPRnFIRI= +cloud.google.com/go/spanner v1.36.0 h1:MYc3fKJlZZCpZymoKBqPR23Hxd1CFhH+zsQPMzeM1xI= +cloud.google.com/go/spanner v1.36.0/go.mod h1:RKVKnqXxTMDuBPAsjxohvcSTH6qiRB6E0oMljFIKPr0= cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos= cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk= @@ -214,6 +216,10 @@ github.com/cenkalti/backoff/v4 v4.1.3 h1:cFAlzYUlVYDysBEH2T5hyJZMh3+5+WCBvSnK6Q8 github.com/cenkalti/backoff/v4 v4.1.3/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInqkPWOWmG2CLw= github.com/census-instrumentation/opencensus-proto v0.2.1 h1:glEXhBS5PSLLv4IXzLA5yPRVX4bilULVyxxbrfOtDAk= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= +github.com/census-instrumentation/opencensus-proto v0.3.0 h1:t/LhUZLVitR1Ow2YOnduCsavhwFUklBMoGVYUCqmCqk= +github.com/census-instrumentation/opencensus-proto v0.3.0/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= +github.com/certifi/gocertifi v0.0.0-20191021191039-0944d244cd40/go.mod h1:sGbDF6GwGcLpkNXPUTkMRoywsNa/ol15pxFe6ERfguA= +github.com/certifi/gocertifi v0.0.0-2020090541-2c3bb06c6054/go.mod h1:sGbDF6GwGcLpkNXPUTkMRoywsNa/ol15pxFe6ERfguA= github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE= diff --git a/sdks/go/pkg/beam/io/spannerio/spanner.go b/sdks/go/pkg/beam/io/spannerio/spanner.go new file mode 100644 index 000..4ad9158a603 --- /dev/null +++ b/sdks/go/pkg/beam/io/spannerio/spanner.go @@ -0,0 +1,241 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +//http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package spannerio provides an API for reading and writing resouces to +// Google Spanner datastores. +package
[beam] branch master updated: [#24266] Update release candidate script to use -PisRelease (#24269)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9da27671cdc [#24266] Update release candidate script to use
-PisRelease (#24269)
9da27671cdc is described below
commit 9da27671cdc8b3df2c548d92a4b2e34f5e0aaa0f
Author: Luke Cwik
AuthorDate: Tue Nov 22 14:35:19 2022 -0800
[#24266] Update release candidate script to use -PisRelease (#24269)
Note that this adds additional tasks which in the past have caused failures
which is why we add --no-daemon and --no-parallel
Fixes #24266
---
release/src/main/scripts/build_release_candidate.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/release/src/main/scripts/build_release_candidate.sh
b/release/src/main/scripts/build_release_candidate.sh
index 0c6da71c76c..6409cf892f1 100755
--- a/release/src/main/scripts/build_release_candidate.sh
+++ b/release/src/main/scripts/build_release_candidate.sh
@@ -344,7 +344,7 @@ if [[ $confirmation = "y" ]]; then
cd ${BEAM_ROOT_DIR}
git checkout ${RC_TAG}
- ./gradlew :pushAllDockerImages -Pdocker-pull-licenses
-Pdocker-tag=${RELEASE}rc${RC_NUM} -Pjava11Home=${JAVA11_HOME}
+ ./gradlew :pushAllDockerImages -PisRelease -Pdocker-pull-licenses
-Pdocker-tag=${RELEASE}rc${RC_NUM} -Pjava11Home=${JAVA11_HOME} --no-daemon
--no-parallel
wipe_local_clone_dir
fi
@@ -383,7 +383,7 @@ if [[ $confirmation = "y" ]]; then
cd ~/${LOCAL_WEBSITE_UPDATE_DIR}/${LOCAL_JAVA_DOC}
git clone --branch "${RC_TAG}" --depth 1 ${GIT_REPO_URL}
cd ${BEAM_ROOT_DIR}
- ./gradlew :sdks:java:javadoc:aggregateJavadoc
+ ./gradlew :sdks:java:javadoc:aggregateJavadoc -PisRelease --no-daemon
--no-parallel
GENERATE_JAVADOC=~/${LOCAL_WEBSITE_UPDATE_DIR}/${LOCAL_JAVA_DOC}/${BEAM_ROOT_DIR}/sdks/java/javadoc/build/docs/javadoc/
echo "--Updating Release Docs-"
[beam] branch asf-site updated: Publishing website 2022/11/22 22:16:55 at commit 67e2008
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new d6df8ed668b Publishing website 2022/11/22 22:16:55 at commit 67e2008 d6df8ed668b is described below commit d6df8ed668b92ea0e392699e7bb0b83ed1df2030 Author: jenkins AuthorDate: Tue Nov 22 22:16:56 2022 + Publishing website 2022/11/22 22:16:55 at commit 67e2008 --- website/generated-content/documentation/index.xml | 117 +++-- .../documentation/ml/anomaly-detection/index.html | 26 ++--- .../documentation/ml/data-processing/index.html| 6 +- .../ml/multi-model-pipelines/index.html| 36 +++ .../documentation/ml/online-clustering/index.html | 16 +-- .../documentation/ml/orchestration/index.html | 35 +++--- .../documentation/ml/overview/index.html | 3 +- .../ml/runinference-metrics/index.html | 6 +- website/generated-content/sitemap.xml | 2 +- 9 files changed, 125 insertions(+), 122 deletions(-) diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml index 9605c47dec9..50e9b3aaf4c 100644 --- a/website/generated-content/documentation/index.xml +++ b/website/generated-content/documentation/index.xml @@ -357,10 +357,10 @@ See the License for the specific language governing permissions and limitations under the License. --> h1 id="anomaly-detection-example">Anomaly Detection Example/h1> -p>The AnomalyDetection example demonstrates how to setup an anomaly detection pipeline that reads text from PubSub in real-time, and then detects anomaly using a trained HDBSCAN clustering model./p> -h3 id="dataset-for-anomaly-detection">Dataset for Anomaly Detection/h3> -p>For the example, we use a dataset called a href="https://huggingface.co/datasets/emotion;>emotion/a>. It comprises of 20,000 English Twitter messages with 6 basic emotions: anger, fear, joy, love, sadness, and surprise. The dataset has three splits: train (for training), validation and test (for performance evaluation). It is a supervised dataset as it contains the text and the category (class) of the dataset. This dataset can easily be accessed using a href="https://hu [...] -p>To have a better understanding of the dataset, here are some examples from the train split of the dataset:/p> +p>The anomaly detection example demonstrates how to set up an anomaly detection pipeline that reads text from Pub/Sub in real time, and then detects anomalies using a trained HDBSCAN clustering model./p> +h2 id="dataset-for-anomaly-detection">Dataset for Anomaly Detection/h2> +p>This example uses a dataset called a href="https://huggingface.co/datasets/emotion;>emotion/a> that contains 20,000 English Twitter messages with 6 basic emotions: anger, fear, joy, love, sadness, and surprise. The dataset has three splits: train (for training), validation, and test (for performance evaluation). Because it contains the text and the category (class) of the dataset, it is a supervised dataset. You can use the a href="https://huggingface.co/docs/datasets/i [...] +p>The following text shows examples from the train split of the dataset:/p> table> thead> tr> @@ -395,12 +395,12 @@ limitations under the License. /tr> /tbody> /table> -h3 id="anomaly-detection-algorithm">Anomaly Detection Algorithm/h3> -p>a href="https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html;>HDBSCAN/a> is a clustering algorithm which extends DBSCAN by converting it into a hierarchical clustering algorithm, and then using a technique to extract a flat clustering based in the stability of clusters. Once trained, the model will predict -1 if a new data point is an outlier, otherwise it will predict one of the existing clusters./p> -h2 id="ingestion-to-pubsub">Ingestion to PubSub/h2> -p>We first ingest the data into a href="https://cloud.google.com/pubsub/docs/overview;>PubSub/a> so that while clustering we can read the tweets from PubSub. PubSub is a messaging service for exchanging event data among applications and services. It is used for streaming analytics and data integration pipelines to ingest and distribute data./p> -p>The full example code for ingesting data to PubSub can be found a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/anomaly_detection/write_data_to_pubsub_pipeline/;>here/a>/p> -p>The file structure for ingestion pipeline is:/p> +h2 id="anomaly-detection-algorithm">Anomaly Detection Algorithm/h2> +p>a href="https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html;>HDBSCAN/a> is a clustering algorithm that extends DBSCAN by converting it into a hierarchical clustering algorithm and then extracting a flat clustering based in the stability of clusters. When trained, the model
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 21645700048 Updating config from bot 21645700048 is described below commit 21645700048ecb8282e01559824d3aa41249fc05 Author: github-actions AuthorDate: Tue Nov 22 21:33:46 2022 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-24315.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24315.json b/scripts/ci/pr-bot/state/pr-state/pr-24315.json index eb0c7805a62..910fcc6b720 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-24315.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-24315.json @@ -7,5 +7,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1219b3b0700 Updating config from bot
1219b3b0700 is described below
commit 1219b3b07007289aa710704610b7ac4476b83f10
Author: github-actions
AuthorDate: Tue Nov 22 21:04:59 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
index 5413e021474..3f9eb6114f0 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
@@ -2,7 +2,7 @@
"label": "io",
"dateOfLastReviewAssignment": {
"chamikaramj": 1669145648882,
-"johnjcasey": 1668814503627,
+"johnjcasey": 1669151092253,
"pabloem": 1669050393201,
"Abacn": 1669057497006,
"ahmedabu98": 1668897324082
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 02e34424b9f Updating config from bot 02e34424b9f is described below commit 02e34424b9f86eb10259ae37cf8838580508dd5d Author: github-actions AuthorDate: Tue Nov 22 21:04:56 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index 05117a479e8..251d749cc3a 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -5,6 +5,6 @@ "kennknowles": 1669145635270, "robertwb": 1669062868143, "kileys": 1669057497006, -"apilloud": 1669050393201 +"apilloud": 1669151092253 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new eba0697e444 Updating config from bot
eba0697e444 is described below
commit eba0697e44470cca54b574d0d844b2ac8d04c0ff
Author: github-actions
AuthorDate: Tue Nov 22 21:04:54 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24315.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24315.json
b/scripts/ci/pr-bot/state/pr-state/pr-24315.json
new file mode 100644
index 000..eb0c7805a62
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24315.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "apilloud",
+"io": "johnjcasey"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 6ba78717597 Updating config from bot 6ba78717597 is described below commit 6ba78717597c4d893dad4e42579db97975b0d1ad Author: github-actions AuthorDate: Tue Nov 22 20:51:35 2022 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-24205.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24205.json b/scripts/ci/pr-bot/state/pr-state/pr-24205.json index 3f69f1bcbe9..114536530f1 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-24205.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-24205.json @@ -5,7 +5,7 @@ "io": "chamikaramj" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": false } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 5609e18ff42 Updating config from bot
5609e18ff42 is described below
commit 5609e18ff42a387dcf81fcc089e2b3b5f2d0d981
Author: github-actions
AuthorDate: Tue Nov 22 20:51:26 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24316.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24316.json
b/scripts/ci/pr-bot/state/pr-state/pr-24316.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24316.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new d3c37d4d54a Updating config from bot d3c37d4d54a is described below commit d3c37d4d54a26aa30eef6fb6eb33cb1649a543e2 Author: github-actions AuthorDate: Tue Nov 22 20:33:41 2022 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-24264.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24264.json b/scripts/ci/pr-bot/state/pr-state/pr-24264.json index 8e49944d7db..67ae83a6cc7 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-24264.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-24264.json @@ -5,7 +5,7 @@ "build": "Abacn" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": false } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new f140f01c91d Updating config from bot
f140f01c91d is described below
commit f140f01c91d1d7598cd3b77cef9c9ee1eb1992e5
Author: github-actions
AuthorDate: Tue Nov 22 19:34:15 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
index 2beff0a8942..5413e021474 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
@@ -1,7 +1,7 @@
{
"label": "io",
"dateOfLastReviewAssignment": {
-"chamikaramj": 1668895469641,
+"chamikaramj": 1669145648882,
"johnjcasey": 1668814503627,
"pabloem": 1669050393201,
"Abacn": 1669057497006,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new e058f45e046 Updating config from bot
e058f45e046 is described below
commit e058f45e04642aa495170aa160024205fe0acc1c
Author: github-actions
AuthorDate: Tue Nov 22 19:34:12 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
index f15c690466f..05117a479e8 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
@@ -1,7 +1,7 @@
{
"label": "java",
"dateOfLastReviewAssignment": {
-"lukecwik": 1669032862586,
+"lukecwik": 1669145648881,
"kennknowles": 1669145635270,
"robertwb": 1669062868143,
"kileys": 1669057497006,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 32bb6421ce6 Updating config from bot
32bb6421ce6 is described below
commit 32bb6421ce69cf14d6182c4428c07f84f10298ab
Author: github-actions
AuthorDate: Tue Nov 22 19:34:10 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24205.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24205.json
b/scripts/ci/pr-bot/state/pr-state/pr-24205.json
new file mode 100644
index 000..3f69f1bcbe9
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24205.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "lukecwik",
+"io": "chamikaramj"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new fe8ab94a716 Updating config from bot
fe8ab94a716 is described below
commit fe8ab94a71632884c0500a67363cae07b018d13e
Author: github-actions
AuthorDate: Tue Nov 22 19:33:59 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
index a1530a964e5..f15c690466f 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
@@ -2,7 +2,7 @@
"label": "java",
"dateOfLastReviewAssignment": {
"lukecwik": 1669032862586,
-"kennknowles": 1669032851444,
+"kennknowles": 1669145635270,
"robertwb": 1669062868143,
"kileys": 1669057497006,
"apilloud": 1669050393201
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 94fac276bdf Updating config from bot
94fac276bdf is described below
commit 94fac276bdf89592ade8e3751d4149d924f60899
Author: github-actions
AuthorDate: Tue Nov 22 19:33:57 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24308.json | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24308.json
b/scripts/ci/pr-bot/state/pr-state/pr-24308.json
index 242a48d7d3b..fa3809f24dd 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-24308.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24308.json
@@ -1,7 +1,9 @@
{
- "commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "kennknowles"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[beam] branch master updated: Add retry to test connections (#23757)
This is an automated email from the ASF dual-hosted git repository.
johncasey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 67e2008c0ee Add retry to test connections (#23757)
67e2008c0ee is described below
commit 67e2008c0ee6878e6a95a361e41d02734fba4ad1
Author: johnjcasey <95318300+johnjca...@users.noreply.github.com>
AuthorDate: Tue Nov 22 13:38:18 2022 -0500
Add retry to test connections (#23757)
* Move connection setup logic for JDBCIO WriteFn to @startBundle to limit
parallel calls to datasource.getConnection()
* move connection setup logic back to processElement.
Put a retry into the DatabaseTestHelper
* run spotless
* use fluent backoff instead of manual implementation
* refactor to manual resource management
* run spotless
---
.../beam/sdk/io/common/DatabaseTestHelper.java | 42 --
1 file changed, 39 insertions(+), 3 deletions(-)
diff --git
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/DatabaseTestHelper.java
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/DatabaseTestHelper.java
index 3cfe08ba4d1..1204fa7ada6 100644
---
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/DatabaseTestHelper.java
+++
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/DatabaseTestHelper.java
@@ -34,8 +34,13 @@ import java.util.Map;
import java.util.Optional;
import java.util.stream.Collectors;
import javax.sql.DataSource;
+import org.apache.beam.sdk.util.BackOff;
+import org.apache.beam.sdk.util.BackOffUtils;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.Sleeper;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
+import org.joda.time.Duration;
import org.postgresql.ds.PGSimpleDataSource;
import org.testcontainers.containers.JdbcDatabaseContainer;
@@ -86,9 +91,40 @@ public class DatabaseTestHelper {
fieldsAndTypes.stream()
.map(kv -> kv.getKey() + " " + kv.getValue())
.collect(Collectors.joining(", "));
-try (Connection connection = dataSource.getConnection()) {
- try (Statement statement = connection.createStatement()) {
-statement.execute(String.format("create table %s (%s)", tableName,
fieldsList));
+SQLException exception = null;
+Sleeper sleeper = Sleeper.DEFAULT;
+BackOff backoff =
+FluentBackoff.DEFAULT
+.withInitialBackoff(Duration.standardSeconds(1))
+.withMaxCumulativeBackoff(Duration.standardMinutes(5))
+.withMaxRetries(4)
+.backoff();
+while (true) {
+ // This is not implemented as try-with-resources because it appears that
try-with-resources is
+ // not correctly catching the PSQLException thrown by
dataSource.getConnection()
+ Connection connection = null;
+ try {
+connection = dataSource.getConnection();
+try (Statement statement = connection.createStatement()) {
+ statement.execute(String.format("create table %s (%s)", tableName,
fieldsList));
+ return;
+}
+ } catch (SQLException e) {
+exception = e;
+ } finally {
+if (connection != null) {
+ connection.close();
+}
+ }
+ boolean hasNext;
+ try {
+hasNext = BackOffUtils.next(sleeper, backoff);
+ } catch (Exception e) {
+throw new RuntimeException(e);
+ }
+ if (!hasNext) {
+// we tried the max number of times
+throw exception;
}
}
}
[beam] branch master updated (883a362c930 -> fe4a98a9b3f)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 883a362c930 Teach Azure Filesystem to authenticate using DefaultAzureCredential in the Python SDK (#24212) add 61e19e4d7ac Fix mvn command to refer the GCP_REGION variable new fe4a98a9b3f Merge pull request #24302: Fix mvn command to refer the GCP_REGION variable The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: examples/multi-language/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #24302: Fix mvn command to refer the GCP_REGION variable
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit fe4a98a9b3f3d1341a2c35a4f262bc8c82307e55 Merge: 883a362c930 61e19e4d7ac Author: Chamikara Jayalath AuthorDate: Tue Nov 22 10:19:23 2022 -0800 Merge pull request #24302: Fix mvn command to refer the GCP_REGION variable examples/multi-language/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated: Teach Azure Filesystem to authenticate using DefaultAzureCredential in the Python SDK (#24212)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 883a362c930 Teach Azure Filesystem to authenticate using
DefaultAzureCredential in the Python SDK (#24212)
883a362c930 is described below
commit 883a362c930aca4298551697d7aaacbe7b6602f1
Author: creste
AuthorDate: Tue Nov 22 12:23:04 2022 -0500
Teach Azure Filesystem to authenticate using DefaultAzureCredential in the
Python SDK (#24212)
* Added support for using DefaultAzureCredential() with Blob Storage.
Tests not working yet due to TLS issue with Azurite.
* SubjectAltNames are now included in the SSL cert.
* DefaultAzureCredential is no longer used with
AZURE_STORAGE_CONNECTION_STRING.
* Fixed integration tests.
* Can now run azure integration tests in parallel across all python
versions.
* Change pipeline option names to be consistent with Java SDK.
* Fix missing Apache 2.0 licenses.
Deleted cert and key files because they can't have a license header.
Moved cert generation to azure_integration_test.sh.
* Fix path to azure integration tests in tox comment.
* Fix path to cert.pem.
* Fix linting issues.
* Fix path to cert.pem
* Fix linter error.
* Added azure integration tests to python 3.7 post-commit test suite.
* Fix unit tests.
* Updated README.md.
* Move changes entry to 2.44.0.
---
CHANGES.md | 1 +
build.gradle.kts | 1 +
sdks/python/apache_beam/internal/azure/__init__.py | 18 +
sdks/python/apache_beam/internal/azure/auth.py | 83 ++
.../apache_beam/io/azure/blobstoragefilesystem.py | 28 +---
.../io/azure/blobstoragefilesystem_test.py | 24 +++
sdks/python/apache_beam/io/azure/blobstorageio.py | 17 -
.../io/azure/integration_test/Dockerfile | 46
.../integration_test/azure_integration_test.sh | 79
.../io/azure/integration_test/docker-compose.yml | 50 +
.../apache_beam/io/azure/integration_test/ssl.conf | 37 ++
.../python/apache_beam/options/pipeline_options.py | 29
sdks/python/setup.py | 1 +
sdks/python/test-suites/direct/common.gradle | 9 +++
sdks/python/tox.ini| 37 ++
15 files changed, 434 insertions(+), 26 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index 414241d1c5b..2446bee91d1 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -61,6 +61,7 @@
* Support for Bigtable sink (Write and WriteBatch) added (Go)
([#23324](https://github.com/apache/beam/issues/23324)).
* S3 implementation of the Beam filesystem (Go)
([#23991](https://github.com/apache/beam/issues/23991)).
* Support for SingleStoreDB source and sink added (Java)
([#22617](https://github.com/apache/beam/issues/22617)).
+* Added support for DefaultAzureCredential authentication in Azure Filesystem
(Python) ([#24210](https://github.com/apache/beam/issues/24210)).
## New Features / Improvements
diff --git a/build.gradle.kts b/build.gradle.kts
index 2556db7bc9a..fe0956a6ea9 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -340,6 +340,7 @@ tasks.register("python37PostCommit") {
dependsOn(":sdks:python:test-suites:direct:py37:postCommitIT")
dependsOn(":sdks:python:test-suites:direct:py37:directRunnerIT")
dependsOn(":sdks:python:test-suites:direct:py37:hdfsIntegrationTest")
+ dependsOn(":sdks:python:test-suites:direct:py37:azureIntegrationTest")
dependsOn(":sdks:python:test-suites:direct:py37:mongodbioIT")
dependsOn(":sdks:python:test-suites:portable:py37:postCommitPy37")
dependsOn(":sdks:python:test-suites:dataflow:py37:spannerioIT")
diff --git a/sdks/python/apache_beam/internal/azure/__init__.py
b/sdks/python/apache_beam/internal/azure/__init__.py
new file mode 100644
index 000..0bce5d68f72
--- /dev/null
+++ b/sdks/python/apache_beam/internal/azure/__init__.py
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific
[beam] branch master updated: TensorRT Custom Inference Function Implementation (#24039)
This is an automated email from the ASF dual-hosted git repository.
jrmccluskey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b5ad9f039ca TensorRT Custom Inference Function Implementation (#24039)
b5ad9f039ca is described below
commit b5ad9f039ca4f70dc2788dad3c2df359ae0405a3
Author: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com>
AuthorDate: Tue Nov 22 12:19:42 2022 -0500
TensorRT Custom Inference Function Implementation (#24039)
* Initial TensorRT implementation
* Formatter
* add unit test
* linting
* duplicate _assign_or_fail logic
---
.../apache_beam/ml/inference/tensorrt_inference.py | 93
.../ml/inference/tensorrt_inference_test.py| 98 ++
2 files changed, 156 insertions(+), 35 deletions(-)
diff --git a/sdks/python/apache_beam/ml/inference/tensorrt_inference.py
b/sdks/python/apache_beam/ml/inference/tensorrt_inference.py
index 8ff65658c6b..5abbe50b329 100644
--- a/sdks/python/apache_beam/ml/inference/tensorrt_inference.py
+++ b/sdks/python/apache_beam/ml/inference/tensorrt_inference.py
@@ -22,6 +22,7 @@ from __future__ import annotations
import logging
import threading
from typing import Any
+from typing import Callable
from typing import Dict
from typing import Iterable
from typing import Optional
@@ -164,11 +165,63 @@ class TensorRTEngine:
self.stream)
+TensorRTInferenceFn = Callable[
+[Sequence[np.ndarray], TensorRTEngine, Optional[Dict[str, Any]]],
+Iterable[PredictionResult]]
+
+
+def _default_tensorRT_inference_fn(
+batch: Sequence[np.ndarray],
+engine: TensorRTEngine,
+inference_args: Optional[Dict[str,
+ Any]] = None) -> Iterable[PredictionResult]:
+ from cuda import cuda
+ (
+ engine,
+ context,
+ context_lock,
+ inputs,
+ outputs,
+ gpu_allocations,
+ cpu_allocations,
+ stream) = engine.get_engine_attrs()
+
+ # Process I/O and execute the network
+ with context_lock:
+_assign_or_fail(
+cuda.cuMemcpyHtoDAsync(
+inputs[0]['allocation'],
+np.ascontiguousarray(batch),
+inputs[0]['size'],
+stream))
+context.execute_async_v2(gpu_allocations, stream)
+for output in range(len(cpu_allocations)):
+ _assign_or_fail(
+ cuda.cuMemcpyDtoHAsync(
+ cpu_allocations[output],
+ outputs[output]['allocation'],
+ outputs[output]['size'],
+ stream))
+_assign_or_fail(cuda.cuStreamSynchronize(stream))
+
+return [
+PredictionResult(
+x, [prediction[idx] for prediction in cpu_allocations]) for idx,
+x in enumerate(batch)
+]
+
+
@experimental(extra_message="No backwards-compatibility guarantees.")
class TensorRTEngineHandlerNumPy(ModelHandler[np.ndarray,
PredictionResult,
TensorRTEngine]):
- def __init__(self, min_batch_size: int, max_batch_size: int, **kwargs):
+ def __init__(
+ self,
+ min_batch_size: int,
+ max_batch_size: int,
+ *,
+ inference_fn: TensorRTInferenceFn = _default_tensorRT_inference_fn,
+ **kwargs):
"""Implementation of the ModelHandler interface for TensorRT.
Example Usage::
@@ -185,6 +238,8 @@ class TensorRTEngineHandlerNumPy(ModelHandler[np.ndarray,
Args:
min_batch_size: minimum accepted batch size.
max_batch_size: maximum accepted batch size.
+ inference_fn: the inference function to use on RunInference calls.
+default: _default_tensorRT_inference_fn
kwargs: Additional arguments like 'engine_path' and 'onnx_path' are
currently supported.
@@ -193,6 +248,7 @@ class TensorRTEngineHandlerNumPy(ModelHandler[np.ndarray,
"""
self.min_batch_size = min_batch_size
self.max_batch_size = max_batch_size
+self.inference_fn = inference_fn
if 'engine_path' in kwargs:
self.engine_path = kwargs.get('engine_path')
elif 'onnx_path' in kwargs:
@@ -241,40 +297,7 @@ class TensorRTEngineHandlerNumPy(ModelHandler[np.ndarray,
Returns:
An Iterable of type PredictionResult.
"""
-from cuda import cuda
-(
-engine,
-context,
-context_lock,
-inputs,
-outputs,
-gpu_allocations,
-cpu_allocations,
-stream) = engine.get_engine_attrs()
-
-# Process I/O and execute the network
-with context_lock:
- _assign_or_fail(
- cuda.cuMemcpyHtoDAsync(
- inputs[0]['allocation'],
- np.ascontiguousarray(batch),
- inputs[0]['size'],
- stream))
- context.execute_async_v2(gpu_allocations, stream)
- for output in
[beam] branch master updated: Copy editing the machine learning pages (#24301)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 22c52d34b3f Copy editing the machine learning pages (#24301) 22c52d34b3f is described below commit 22c52d34b3faff66209acde13e65d7fcca791865 Author: Rebecca Szper <98840847+rsz...@users.noreply.github.com> AuthorDate: Tue Nov 22 08:43:02 2022 -0800 Copy editing the machine learning pages (#24301) * copy editing the AI/ML pages * copy editing the AI/ML pages * copy editing the AI/ML pages * copy editing the AI/ML pages * copy editing the AI/ML pages * Update website/www/site/content/en/documentation/ml/data-processing.md Co-authored-by: Danny McCormick * Update website/www/site/content/en/documentation/ml/online-clustering.md Co-authored-by: Danny McCormick * Update website/www/site/content/en/documentation/ml/online-clustering.md Co-authored-by: Danny McCormick * copy editing the AI/ML pages * trim trailing whitespace Co-authored-by: Danny McCormick --- .../en/documentation/ml/anomaly-detection.md | 117 ++-- .../content/en/documentation/ml/data-processing.md | 34 +++--- .../en/documentation/ml/multi-model-pipelines.md | 38 +++ .../en/documentation/ml/online-clustering.md | 121 +++-- .../content/en/documentation/ml/orchestration.md | 94 +--- .../site/content/en/documentation/ml/overview.md | 63 ++- .../en/documentation/ml/runinference-metrics.md| 54 - 7 files changed, 274 insertions(+), 247 deletions(-) diff --git a/website/www/site/content/en/documentation/ml/anomaly-detection.md b/website/www/site/content/en/documentation/ml/anomaly-detection.md index 4f98ff5481b..c37a032cdd9 100644 --- a/website/www/site/content/en/documentation/ml/anomaly-detection.md +++ b/website/www/site/content/en/documentation/ml/anomaly-detection.md @@ -17,13 +17,13 @@ limitations under the License. # Anomaly Detection Example -The AnomalyDetection example demonstrates how to setup an anomaly detection pipeline that reads text from PubSub in real-time, and then detects anomaly using a trained HDBSCAN clustering model. +The anomaly detection example demonstrates how to set up an anomaly detection pipeline that reads text from Pub/Sub in real time, and then detects anomalies using a trained HDBSCAN clustering model. -### Dataset for Anomaly Detection -For the example, we use a dataset called [emotion](https://huggingface.co/datasets/emotion). It comprises of 20,000 English Twitter messages with 6 basic emotions: anger, fear, joy, love, sadness, and surprise. The dataset has three splits: train (for training), validation and test (for performance evaluation). It is a supervised dataset as it contains the text and the category (class) of the dataset. This dataset can easily be accessed using [HuggingFace Datasets](https://huggingface.co [...] +## Dataset for Anomaly Detection +This example uses a dataset called [emotion](https://huggingface.co/datasets/emotion) that contains 20,000 English Twitter messages with 6 basic emotions: anger, fear, joy, love, sadness, and surprise. The dataset has three splits: train (for training), validation, and test (for performance evaluation). Because it contains the text and the category (class) of the dataset, it is a supervised dataset. You can use the [Hugging Face datasets page](https://huggingface.co/docs/datasets/index) [...] -To have a better understanding of the dataset, here are some examples from the train split of the dataset: +The following text shows examples from the train split of the dataset: | Text| Type of emotion | @@ -35,16 +35,16 @@ To have a better understanding of the dataset, here are some examples from the t | i feel you know basically like a fake in the realm of science fiction | Sadness | | i began having them several times a week feeling tortured by the hallucinations moving people and figures sounds and vibrations | Fear | -### Anomaly Detection Algorithm -[HDBSCAN](https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html) is a clustering algorithm which extends DBSCAN by converting it into a hierarchical clustering algorithm, and then using a technique to extract a flat clustering based in the stability of clusters. Once trained, the model will predict -1 if a new data point is an outlier, otherwise it will predict one of the existing clusters. +## Anomaly Detection Algorithm +[HDBSCAN](https://hdbscan.readthedocs.io/en/latest/how_hdbscan_works.html) is a clustering algorithm that extends DBSCAN by converting it into a hierarchical clustering algorithm and then extracting a flat clustering based in the stability of clusters. When
[beam] branch users/damccorm/flink-fixes updated (71218732dc0 -> c2b58515b44)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/flink-fixes in repository https://gitbox.apache.org/repos/asf/beam.git from 71218732dc0 Try adding option to flinkOptions add c2b58515b44 Try manually setting dir No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/portability/flink_runner.py | 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new b12a6edcd74 Updating config from bot b12a6edcd74 is described below commit b12a6edcd7447f667e9a5c60498b2d70c76568db Author: github-actions AuthorDate: Tue Nov 22 16:05:17 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json b/scripts/ci/pr-bot/state/reviewers-for-label-go.json index 1804612e6e6..8e01b316bd4 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json @@ -5,6 +5,6 @@ "jrmccluskey": 1669008889648, "youngoli": 1657688896155, "damccorm": 1669007109184, -"riteshghorse": 1668746356305 +"riteshghorse": 1669133112874 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 4c96da4cfd7 Updating config from bot
4c96da4cfd7 is described below
commit 4c96da4cfd77d01df0ad2edcd9182fd92afc4b3a
Author: github-actions
AuthorDate: Tue Nov 22 16:05:14 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24311.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24311.json
b/scripts/ci/pr-bot/state/pr-state/pr-24311.json
new file mode 100644
index 000..aa3638a74ec
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24311.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "riteshghorse"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch users/damccorm/flink-fixes updated (2c82146993a -> 71218732dc0)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/flink-fixes in repository https://gitbox.apache.org/repos/asf/beam.git from 2c82146993a Try original fix with updated paths add 71218732dc0 Try adding option to flinkOptions No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/options/pipeline_options.py | 1 + sdks/python/apache_beam/runners/portability/flink_runner.py | 6 +- sdks/python/test-suites/portable/common.gradle | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-)
[beam] branch dependabot/go_modules/sdks/cloud.google.com/go/bigquery-1.43.0 created (now 4da135257e9)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/bigquery-1.43.0 in repository https://gitbox.apache.org/repos/asf/beam.git at 4da135257e9 Bump cloud.google.com/go/bigquery from 1.42.0 to 1.43.0 in /sdks No new revisions were added by this update.
[beam] branch master updated: Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#24280)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 88c1f0a486b Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#24280) 88c1f0a486b is described below commit 88c1f0a486b984bb20abd225e605ac4292aef604 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Nov 22 10:14:34 2022 -0500 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#24280) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.11.39 to 1.11.41. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/manager/v1.11.39...feature/s3/manager/v1.11.41) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 8 ++-- 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index bf6108338fe..061250fc0dd 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -31,7 +31,7 @@ require ( github.com/aws/aws-sdk-go-v2 v1.17.1 github.com/aws/aws-sdk-go-v2/config v1.18.2 github.com/aws/aws-sdk-go-v2/credentials v1.13.2 - github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.39 + github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.41 github.com/aws/aws-sdk-go-v2/service/s3 v1.29.3 github.com/aws/smithy-go v1.13.4 github.com/docker/go-connections v0.4.0 diff --git a/sdks/go.sum b/sdks/go.sum index 19eb841e162..acee0826903 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -149,19 +149,17 @@ github.com/aws/aws-sdk-go-v2 v1.17.1/go.mod h1:JLnGeGONAyi2lWXI1p0PCIOIy333JMVK1 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.9 h1:RKci2D7tMwpvGpDNZnGQw9wk6v7o/xSwFcUAuNPoB8k= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.9/go.mod h1:vCmV1q1VK8eoQJ5+aYE7PkK1K6v41qJ5pJdK3ggCDvg= github.com/aws/aws-sdk-go-v2/config v1.5.0/go.mod h1:RWlPOAW3E3tbtNAqTwvSW54Of/yP3oiZXMI0xfUdjyA= -github.com/aws/aws-sdk-go-v2/config v1.18.0/go.mod h1:H13DRX9Nv5tAcQvPABrE3dm5XnLp1RC7fVSM3OWiLvA= github.com/aws/aws-sdk-go-v2/config v1.18.2 h1:tRhTb3xMZsB0gW0sXWpqs9FeIP8iQp5SvnvwiPXzHwo= github.com/aws/aws-sdk-go-v2/config v1.18.2/go.mod h1:9XVoZTdD8ICjrgI5ddb8j918q6lEZkFYpb7uohgvU6c= github.com/aws/aws-sdk-go-v2/credentials v1.3.1/go.mod h1:r0n73xwsIVagq8RsxmZbGSRQFj9As3je72C2WzUIToc= -github.com/aws/aws-sdk-go-v2/credentials v1.13.0/go.mod h1:prZpUfBu1KZLBLVX482Sq4DpDXGugAre08TPEc21GUg= github.com/aws/aws-sdk-go-v2/credentials v1.13.2 h1:F/v1w0XcFDZjL0bCdi9XWJenoPKjGbzljBhDKcryzEQ= github.com/aws/aws-sdk-go-v2/credentials v1.13.2/go.mod h1:eAT5aj/WJ2UDIA0IVNFc2byQLeD89SDEi4cjzH/MKoQ= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.3.0/go.mod h1:2LAuqPx1I6jNfaGDucWfA2zqQCYCOMCDHiCOciALyNw= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.19 h1:E3PXZSI3F2bzyj6XxUXdTIfvp425HHhwKsFvmzBwHgs= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.19/go.mod h1:VihW95zQpeKQWVPGkwT+2+WJNQV8UXFfMTWdU6VErL8= github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.3.2/go.mod h1:qaqQiHSrOUVOfKe6fhgQ6UzhxjwqVW8aHNegd6Ws4w4= -github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.39 h1:Fz/t08vTFdz63ZnlrQBLOMgBCNqdKmMQWE/XjKm1jt4= -github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.39/go.mod h1:763L1Xloj/mjT0do5k9d0qh0vEAVuomDPn1iOG2AdB4= +github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.41 h1:ssgdsNm11dvFtO7F/AeiW4dAO3eGsDeg5fwpag/JP/I= +github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.41/go.mod h1:CS+AbDFAaPU9TQOo7U6mVV23YvqCOElnqmh0XQjgJ1g= github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.25 h1:nBO/RFxeq/IS5G9Of+ZrgucRciie2qpLy++3UGZ+q2E= github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.25/go.mod h1:Zb29PYkf42vVYQY6pvSyJCJcFHlPIiY+YKdPtwnvMkY= github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.19 h1:oRHDrwCTVT8ZXi4sr9Ld+EXk7N/KGssOr2ygNeojEhw= @@ -183,7 +181,6 @@ github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.5.1/go.mod h1:6EQZIwNN github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.19 h1:piDBAaWkaxkkVV3xJJbTehXCZRXYs49kvpi/LG6LR2o= github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.19/go.mod h1:BmQWRVkLTmyNzYPFAZgon53qKLWBNSvonugD1MrSWUs= github.com/aws/aws-sdk-go-v2/service/s3 v1.11.1/go.mod h1:XLAGFrEjbvMCLvAtWLLP32yTv8GpBquCApZEycDLunI=
[beam] branch asf-site updated: Publishing website 2022/11/22 13:12:18 at commit 4d5937b
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 03a80f01396 Publishing website 2022/11/22 13:12:18 at commit 4d5937b
03a80f01396 is described below
commit 03a80f0139627311c4dee690ebd24ce439834ce4
Author: jenkins
AuthorDate: Tue Nov 22 13:12:19 2022 +
Publishing website 2022/11/22 13:12:18 at commit 4d5937b
---
website/generated-content/case-studies/index.html | 14 +++---
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/website/generated-content/case-studies/index.html
b/website/generated-content/case-studies/index.html
index 9f953433811..11166a837f4 100644
--- a/website/generated-content/case-studies/index.html
+++ b/website/generated-content/case-studies/index.html
@@ -20,13 +20,13 @@ function
addPlaceholder(){$('input:text').attr('placeholder',"What are you looki
function endSearch(){var
search=document.querySelector(".searchBar");search.classList.add("disappear");var
icons=document.querySelector("#iconsBar");icons.classList.remove("disappear");}
function blockScroll(){$("body").toggleClass("fixedPosition");}
function openMenu(){addPlaceholder();blockScroll();}Case
StudiesApache Beam powers many of today’s
leading projects, industry-specific use cases, and
-startups.Powering Streaming and Real-time ML at
IntuitWe feel that the runner
agnosticism of Apache Beam affords flexibility and future-proofs our Stream
Processing Platform as new runtimes are developed. Apache Beam enabled the
democratization of stream processing at Intui [...]
-Real-time ML with Beam at LyftLyft Marketplace team aims to improve our
business efficiency by being nimble to real-world dynamics. Apache Beam has
enabled us to meet the goal of having a robust and scalable ML infrastructure
for improvi [...]
-Real-time Event Stream Processing at Scale for Palo
Alto NetworksPalo Alto Networks is a
global cybersecurity leader that deals with processing hundreds of billions of
security events per day in real-time, which is on the high end of [...]
-Visual Apache Beam Pipeline Design and
Orchestration with Apache HopApache
Hop is an open source data orchestration and engineering platform that extends
Apache Beam with visual pipeline lifecycle management. Neo4j’s Chief Solution
Archit [...]
-Scalability and Cost Optimization for Search
Engine's WorkloadsDive into the Czech
search engine’s experience of scaling the on-premises infrastructure to learn
more about the benefits of byte-based data shuffling and the use cases whe [...]
-Four Apache Technologies Combined for Fun and
ProfitRicardo, the largest online
marketplace in Switzerland, uses Apache Beam to stream-process platform data
and enables the Data Intelligence team to provide scalable data integration,
[...]
-mailto:d...@beam.apache.org?subject=Beam Website Add New Case Study"
class=case-study-primary-button>Share your storyAlso used byhttps://www.zillow.com/ targe [...]
+startups.Powering Streaming and
Real-time ML at IntuitWe feel that
the runner agnosticism of Apache Beam affords flexibility and future-proofs our
Stream Processing Platform as new runtimes are developed. Apache Beam enabled
the democratization of stream proces [...]
+Real-time ML with Beam at LyftLyft Marketplace team aims to improve our
business efficiency by being nimble to real-world dynamics. Apache Beam has
enabled us to meet the goal of having a robust and scalable ML infrastructur
[...]
+Real-time Event Stream Processing at Scale for Palo
Alto NetworksPalo Alto Networks is a
global cybersecurity leader that deals with processing hundreds of billions of
security events per day in real-time, which is on th [...]
+Visual Apache Beam Pipeline Design and
Orchestration with Apache HopApache
Hop is an open source data orchestration and engineering platform that extends
Apache Beam with visual pipeline lifecycle management. Neo4j’s Chief So [...]
+Scalability and Cost Optimization for Search
Engine's WorkloadsDive into the Czech
search engine’s experience of scaling the on-premises infrastructure to learn
more about the benefits of byte-based data shuffling and the [...]
+Four Apache Technologies Combined for Fun and
ProfitRicardo, the largest online
marketplace in Switzerland, uses Apache Beam to stream-process platform data
and enables the Data Intelligence team to provide scalable data [...]
+mailto:d...@beam.apache.org?subject=Beam Website Add New Case Study"
class=case-study-primary-button>Share your storyAlso used byhttps://www.zillow.com/ targe [...]
Add your logohttps://www.apache.org>The Apache Software Foundation
| Privacy Policy
[beam] 01/01: Merge pull request #24251: [Website] add lazy loading attr to images
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 4d5937b9a3d268f6ec7ce919ca9cef8d1db994b3 Merge: 79b8c154b58 be346e619b0 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Nov 22 10:58:21 2022 +0100 Merge pull request #24251: [Website] add lazy loading attr to images website/www/site/layouts/case-studies/list.html | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated (79b8c154b58 -> 4d5937b9a3d)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 79b8c154b58 Merge pull request #24023 Add a Dataflow smoke test. add be346e619b0 [Website] add lazy loading attr to images #24250 new 4d5937b9a3d Merge pull request #24251: [Website] add lazy loading attr to images The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/layouts/case-studies/list.html | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 226e9253fca Updating config from bot
226e9253fca is described below
commit 226e9253fca4eeab5410f99ea182e6b78fa31cf1
Author: github-actions
AuthorDate: Tue Nov 22 09:04:59 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24287.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24287.json
b/scripts/ci/pr-bot/state/pr-state/pr-24287.json
new file mode 100644
index 000..8a92900947e
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24287.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new e09b199c80b Updating config from bot
e09b199c80b is described below
commit e09b199c80b4b627539c2e075af031911406bb95
Author: github-actions
AuthorDate: Tue Nov 22 09:05:03 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index ebd40e4338e..a47920d7f4b 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1669032851728,
+"damccorm": 1669107898121,
"Abacn": 1669062868143
}
}
\ No newline at end of file
