[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 624172b2236 Updating config from bot 624172b2236 is described below commit 624172b2236cecc51bb252b922821f41440ff0d4 Author: github-actions AuthorDate: Tue Dec 6 05:04:51 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index c62e600e1f2..be5cb4b4694 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -5,7 +5,7 @@ "yeandy": 1665802753763, "TheNeuralBit": 1667896849319, "ryanthompson591": 1670002443548, -"tvalentyn": 1669983222906, +"tvalentyn": 1670303087352, "pabloem": 1670287741806, "y1chi": 1667002607045, "damccorm": 1670261644639,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 714dab0123a Updating config from bot
714dab0123a is described below
commit 714dab0123ac7e9034b44dee5b9182b403178e3f
Author: github-actions
AuthorDate: Tue Dec 6 05:04:49 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24539.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24539.json
b/scripts/ci/pr-bot/state/pr-state/pr-24539.json
new file mode 100644
index 000..321b31c0f05
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24539.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "tvalentyn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch nightly-refs/heads/master updated (d15913b2891 -> ad92087babb)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from d15913b2891 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in
/sdks (#24501)
add a9dcf95453d [CdapIO] Add readme for CdapIO. Update readme for
SparkReceiverIO. (#23959)
add c197579e9f2 [Spark Dataset runner] Add @Experimental and reduce
visibility where possible.
add 66db2d81811 Merge pull request #24522: [Spark Dataset runner] Add
@Experimental and reduce visibility where possible
add 54f579e45ab Fix grafana dashboard id (#24524)
add dd8c035937a [Spark runner] Support running (VR) tests with Java 17
(closes #24400) (#24401)
add 12d0ba99d19 Replaced deprecated finalize with DoFn Teardown (#24516)
add 719e5946420 Bump cloud.google.com/go/storage from 1.28.0 to 1.28.1 in
/sdks (#24517)
add a88956b1e7c add clarifier to error message (#24449)
add 4a45066e653 Batch rename requests in fileio.WriteToFiles (#24341)
add 7701db80691 Bump golang.org/x/text from 0.4.0 to 0.5.0 in /sdks
(#24520)
add 7e6e13e5864 Update nbconvert requirement in /sdks/python
add 3e4051cd005 Merge pull request #23240 from
apache/dependabot/pip/sdks/python/nbconvert-gte-6.2.0-and-lt-8
add dbb584997c7 Support for JsonSchema in Kafka Read Schema Transform
(#24272)
add 33af390d3a4 Run go fmt over full go directory with go 1.19 (#24525)
add 9b3b403ae9c Cloudbuild+manualsetup+playground (#24144)
add 6ba427ea6a1 Bump golang.org/x/sys from 0.2.0 to 0.3.0 in /sdks (#24519)
add 9f2de2ab8ed Bump cloud.google.com/go/bigtable from 1.18.0 to 1.18.1 in
/sdks (#24500)
add d793631cd9c Update from interface{} -> any for core packages (#24505)
add cb329e410e9 Implement FileWriteSchemaTransformConfiguration (#24479)
add 92a65cbcf91 Bump cloud.google.com/go/pubsub from 1.27.0 to 1.27.1 in
/sdks (#24518)
add 60aba30e12d [Playground] Healthcheck was added (#24227)
add ad92087babb Update dataflow container version for Pandas upgrade
(#24532)
No new revisions were added by this update.
Summary of changes:
.../Python_ML_RunInference_Benchmark_Tests.json| 2 +-
.../cloudbuild/cloudbuild_pg_infra.yaml| 45 ++
.../cloudbuild/cloudbuild_pg_to_gke.yaml | 48 ++
playground/infrastructure/cloudbuild/env_init.sh | 44 ++
.../autoscaling-frontend.yaml} | 30 +-
.../{values.yaml => templates/autoscaling-go.yaml} | 30 +-
.../autoscaling-java.yaml} | 30 +-
.../autoscaling-python.yaml} | 30 +-
.../autoscaling-router.yaml} | 30 +-
.../autoscaling-scio.yaml} | 30 +-
.../templates/deployment-frontend.yml | 19 +
.../helm-playground/templates/deployment-go.yml| 21 +
.../helm-playground/templates/deployment-java.yml | 21 +
.../templates/deployment-python.yml| 21 +
.../templates/deployment-router.yml| 21 +
.../helm-playground/templates/deployment-scio.yml | 21 +
.../infrastructure/helm-playground/values.yaml | 11 +-
playground/terraform/README.md | 1 +
playground/terraform/build.gradle.kts | 16 +-
.../cloudbuild-manual-setup/01.setup/iam.tf| 44 ++
.../cloudbuild-manual-setup/01.setup/provider.tf | 20 +
.../cloudbuild-manual-setup/01.setup/services.tf | 33 +
.../cloudbuild-manual-setup/01.setup/terraform.tf | 28 +
.../cloudbuild-manual-setup/01.setup/variables.tf | 27 +
.../02.builders/provider.tf| 20 +
.../02.builders/terraform.tf | 28 +
.../02.builders/triggers.tf| 81 +++
.../02.builders/variables.tf | 108
.../cloudbuild-manual-setup/README.md | 116
.../SparkStructuredStreamingPipelineOptions.java | 2 +
.../SparkStructuredStreamingRunner.java| 2 +
.../translation/batch/Aggregators.java | 8 +-
.../batch/CombineGloballyTranslatorBatch.java | 2 +-
.../batch/CombineGroupedValuesTranslatorBatch.java | 2 +-
.../batch/GroupByKeyTranslatorBatch.java | 4 +-
.../translation/batch/ImpulseTranslatorBatch.java | 3 +-
.../batch/ReshuffleTranslatorBatch.java| 4 +-
runners/spark/spark_runner.gradle | 30 +-
sdks/go.mod| 10 +-
sdks/go.sum| 20 +-
sdks/go/cmd/starcgen/starcgen.go | 15 +-
sdks/go/examples/cookbook/tornadoes/tornadoes.go | 4 +-
.../debugging_wordcount/debugging_wordcount.go | 8 +-
sdks/go/examples/fhirio/import/import.go | 33 +-
.../fhirio/read_write_pubsub/read_write_pubsub.go | 37 +-
sdks/go/examples/kafka/taxi.go
[beam] branch dependabot/pip/sdks/python/google-cloud-bigquery-storage-gte-2.6.3-and-lt-2.17 created (now e40c7fd8241)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/pip/sdks/python/google-cloud-bigquery-storage-gte-2.6.3-and-lt-2.17 in repository https://gitbox.apache.org/repos/asf/beam.git at e40c7fd8241 Update google-cloud-bigquery-storage requirement in /sdks/python No new revisions were added by this update.
[beam] branch master updated: Update dataflow container version for Pandas upgrade (#24532)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new ad92087babb Update dataflow container version for Pandas upgrade (#24532) ad92087babb is described below commit ad92087babbe961770d4aafd364cdfb57a2635e5 Author: Danny McCormick AuthorDate: Mon Dec 5 20:13:07 2022 -0500 Update dataflow container version for Pandas upgrade (#24532) --- sdks/python/apache_beam/runners/dataflow/internal/names.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py b/sdks/python/apache_beam/runners/dataflow/internal/names.py index 34ef156415c..9bd9bf55057 100644 --- a/sdks/python/apache_beam/runners/dataflow/internal/names.py +++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py @@ -36,10 +36,10 @@ SERIALIZED_SOURCE_KEY = 'serialized_source' # Update this version to the next version whenever there is a change that will # require changes to legacy Dataflow worker execution environment. -BEAM_CONTAINER_VERSION = 'beam-master-20221122' +BEAM_CONTAINER_VERSION = 'beam-master-20221205' # Update this version to the next version whenever there is a change that # requires changes to SDK harness container or SDK harness launcher. -BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20221122' +BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20221205' DATAFLOW_CONTAINER_IMAGE_REPOSITORY = 'gcr.io/cloud-dataflow/v1beta3'
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 01593bd4f07 Updating config from bot 01593bd4f07 is described below commit 01593bd4f07ea9b439dac72c04135234d2bbb31d Author: github-actions AuthorDate: Tue Dec 6 00:49:07 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index f80c1766766..c62e600e1f2 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -6,7 +6,7 @@ "TheNeuralBit": 1667896849319, "ryanthompson591": 1670002443548, "tvalentyn": 1669983222906, -"pabloem": 1669983210403, +"pabloem": 1670287741806, "y1chi": 1667002607045, "damccorm": 1670261644639, "jrmccluskey": 1670268830062
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 029a8bfa842 Updating config from bot
029a8bfa842 is described below
commit 029a8bfa8424e6bc46b3d3467e64aeab6216d76c
Author: github-actions
AuthorDate: Tue Dec 6 00:49:09 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
index 914c5100414..6251967efb9 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
@@ -2,7 +2,7 @@
"label": "io",
"dateOfLastReviewAssignment": {
"chamikaramj": 1670099635969,
-"johnjcasey": 1670026448158,
+"johnjcasey": 1670287741806,
"pabloem": 1670088828000,
"Abacn": 1670242421955,
"ahmedabu98": 1670092513730
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new bc9bbb67907 Updating config from bot
bc9bbb67907 is described below
commit bc9bbb67907e6ef5f578dddc3a1213c4f07f666e
Author: github-actions
AuthorDate: Tue Dec 6 00:49:03 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24536.json | 11 +++
1 file changed, 11 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24536.json
b/scripts/ci/pr-bot/state/pr-state/pr-24536.json
new file mode 100644
index 000..1508b4d8d6f
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24536.json
@@ -0,0 +1,11 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "pabloem",
+"io": "johnjcasey"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new f02f49990db Updating config from bot
f02f49990db is described below
commit f02f49990db80afcc289704472e6d4bf2006a2c2
Author: github-actions
AuthorDate: Tue Dec 6 00:31:54 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-23988.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-23988.json
b/scripts/ci/pr-bot/state/pr-state/pr-23988.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-23988.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 95799632e2c Updating config from bot 95799632e2c is described below commit 95799632e2cfef1864515de5d973afff9b63782f Author: github-actions AuthorDate: Tue Dec 6 00:16:19 2022 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-24359.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24359.json b/scripts/ci/pr-bot/state/pr-state/pr-24359.json index 063ce8593a5..a7b6e8a2482 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-24359.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-24359.json @@ -5,7 +5,7 @@ "io": "chamikaramj" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": false } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 210d23182ee Updating config from bot 210d23182ee is described below commit 210d23182ee9f2bc4b505456a1c12d1afa0e3ff9 Author: github-actions AuthorDate: Tue Dec 6 00:14:25 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-go.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-go.json b/scripts/ci/pr-bot/state/reviewers-for-label-go.json index 4b847208dc3..640e5315890 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-go.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-go.json @@ -5,6 +5,6 @@ "jrmccluskey": 1670216679908, "youngoli": 1657688896155, "damccorm": 1670216690795, -"riteshghorse": 1670216673847 +"riteshghorse": 1670285661635 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new e02d53b75d3 Updating config from bot
e02d53b75d3 is described below
commit e02d53b75d3ae101447c4272a895b8568680dd15
Author: github-actions
AuthorDate: Tue Dec 6 00:14:23 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24534.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24534.json
b/scripts/ci/pr-bot/state/pr-state/pr-24534.json
new file mode 100644
index 000..aa3638a74ec
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24534.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"go": "riteshghorse"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: [Playground] Healthcheck was added (#24227)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 60aba30e12d [Playground] Healthcheck was added (#24227)
60aba30e12d is described below
commit 60aba30e12d8235510f95c015459815a3a40b747
Author: MakarkinSAkvelon <67736809+makarkinsakve...@users.noreply.github.com>
AuthorDate: Tue Dec 6 05:09:13 2022 +0500
[Playground] Healthcheck was added (#24227)
* Healthcheck was added
* Autoscaling
* License information was added
* Move healthcheck vars to value.yaml
* remove unused values
* Resources limits were corrected
Co-authored-by: Sergey Makarkin
---
.../autoscaling-frontend.yaml} | 30 +-
.../{values.yaml => templates/autoscaling-go.yaml} | 30 +-
.../autoscaling-java.yaml} | 30 +-
.../autoscaling-python.yaml} | 30 +-
.../autoscaling-router.yaml} | 30 +-
.../autoscaling-scio.yaml} | 30 +-
.../templates/deployment-frontend.yml | 19 ++
.../helm-playground/templates/deployment-go.yml| 21 +++
.../helm-playground/templates/deployment-java.yml | 21 +++
.../templates/deployment-python.yml| 21 +++
.../templates/deployment-router.yml| 21 +++
.../helm-playground/templates/deployment-scio.yml | 21 +++
.../infrastructure/helm-playground/values.yaml | 11
13 files changed, 202 insertions(+), 113 deletions(-)
diff --git a/playground/infrastructure/helm-playground/values.yaml
b/playground/infrastructure/helm-playground/templates/autoscaling-frontend.yaml
similarity index 62%
copy from playground/infrastructure/helm-playground/values.yaml
copy to
playground/infrastructure/helm-playground/templates/autoscaling-frontend.yaml
index 96ee08fa7c0..6fc40394330 100644
--- a/playground/infrastructure/helm-playground/values.yaml
+++
b/playground/infrastructure/helm-playground/templates/autoscaling-frontend.yaml
@@ -12,21 +12,15 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-replicaCount: 1
-image:
- java_image: beam_playground-backend-java
- go_image: beam_playground-backend-go
- router_image: beam_playground-backend-router
- scio_image: beam_playground-backend-scio
- python_image: beam_playground-backend-python
- frontend_image: beam_playground-frontend
- pullPolicy: Always
-service:
- type: NodePort
- targetPort: 8080
- port: 443
-autoscaling:
- enabled: false
- minReplicas: 1
- maxReplicas: 4
- targetCPUUtilizationPercentage: 80
\ No newline at end of file
+apiVersion: autoscaling/v1
+kind: HorizontalPodAutoscaler
+metadata:
+ name: playground-frontend
+spec:
+ maxReplicas: 10
+ minReplicas: 1
+ scaleTargetRef:
+apiVersion: apps/v1
+kind: Deployment
+name: playground-frontend
+ targetCPUUtilizationPercentage: 90
diff --git a/playground/infrastructure/helm-playground/values.yaml
b/playground/infrastructure/helm-playground/templates/autoscaling-go.yaml
similarity index 62%
copy from playground/infrastructure/helm-playground/values.yaml
copy to playground/infrastructure/helm-playground/templates/autoscaling-go.yaml
index 96ee08fa7c0..81a28e2d59a 100644
--- a/playground/infrastructure/helm-playground/values.yaml
+++ b/playground/infrastructure/helm-playground/templates/autoscaling-go.yaml
@@ -12,21 +12,15 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-replicaCount: 1
-image:
- java_image: beam_playground-backend-java
- go_image: beam_playground-backend-go
- router_image: beam_playground-backend-router
- scio_image: beam_playground-backend-scio
- python_image: beam_playground-backend-python
- frontend_image: beam_playground-frontend
- pullPolicy: Always
-service:
- type: NodePort
- targetPort: 8080
- port: 443
-autoscaling:
- enabled: false
- minReplicas: 1
- maxReplicas: 4
- targetCPUUtilizationPercentage: 80
\ No newline at end of file
+apiVersion: autoscaling/v1
+kind: HorizontalPodAutoscaler
+metadata:
+ name: playground-go
+spec:
+ maxReplicas: 10
+ minReplicas: 1
+ scaleTargetRef:
+apiVersion: apps/v1
+kind: Deployment
+name: playground-go
+ targetCPUUtilizationPercentage: 90
diff --git a/playground/infrastructure/helm-playground/values.yaml
b/playground/infrastructure/helm-playground/templates/autoscaling-java.yaml
similarity ind
[beam] branch dependabot/go_modules/sdks/cloud.google.com/go/pubsub-1.28.0 created (now 93c53e93f21)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/pubsub-1.28.0 in repository https://gitbox.apache.org/repos/asf/beam.git at 93c53e93f21 Bump cloud.google.com/go/pubsub from 1.27.1 to 1.28.0 in /sdks No new revisions were added by this update.
[beam] branch master updated (cb329e410e9 -> 92a65cbcf91)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from cb329e410e9 Implement FileWriteSchemaTransformConfiguration (#24479) add 92a65cbcf91 Bump cloud.google.com/go/pubsub from 1.27.0 to 1.27.1 in /sdks (#24518) No new revisions were added by this update. Summary of changes: sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new ffca8c85535 Updating config from bot ffca8c85535 is described below commit ffca8c85535863c99bd0dbd3b66bc178252a89e2 Author: github-actions AuthorDate: Mon Dec 5 22:04:43 2022 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-24358.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24358.json b/scripts/ci/pr-bot/state/pr-state/pr-24358.json index 8a92900947e..c82cdd128d1 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-24358.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-24358.json @@ -6,5 +6,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 8b82c66f6c0 Updating config from bot 8b82c66f6c0 is described below commit 8b82c66f6c08248862ba0abfa18ed365890b5f80 Author: github-actions AuthorDate: Mon Dec 5 20:37:12 2022 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-24530.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24530.json b/scripts/ci/pr-bot/state/pr-state/pr-24530.json index 959c289c017..30d25f1f3dd 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-24530.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-24530.json @@ -4,7 +4,7 @@ "python": "jrmccluskey" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": false } \ No newline at end of file
[beam] branch master updated (d793631cd9c -> cb329e410e9)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from d793631cd9c Update from interface{} -> any for core packages (#24505)
add cb329e410e9 Implement FileWriteSchemaTransformConfiguration (#24479)
No new revisions were added by this update.
Summary of changes:
.../java/io/fileschematransform}/OWNERS| 2 +-
.../build.gradle | 33 +--
.../FileWriteSchemaTransformConfiguration.java | 245 +
.../FileWriteSchemaTransformFormatProvider.java} | 21 +-
.../FileWriteSchemaTransformFormatProviders.java | 124 +++
.../sdk/io/fileschematransform}/package-info.java | 4 +-
...leWriteSchemaTransformFormatProvidersTest.java} | 26 ++-
settings.gradle.kts| 1 +
8 files changed, 419 insertions(+), 37 deletions(-)
copy {runners/google-cloud-dataflow-java =>
sdks/java/io/fileschematransform}/OWNERS (84%)
copy sdks/java/io/{contextualtextio => fileschematransform}/build.gradle (60%)
create mode 100644
sdks/java/io/fileschematransform/src/main/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformConfiguration.java
copy
sdks/java/{core/src/main/java/org/apache/beam/sdk/schemas/io/GenericDlqProvider.java
=>
io/fileschematransform/src/main/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProvider.java}
(61%)
create mode 100644
sdks/java/io/fileschematransform/src/main/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProviders.java
copy
{examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/options
=>
sdks/java/io/fileschematransform/src/main/java/org/apache/beam/sdk/io/fileschematransform}/package-info.java
(85%)
copy
sdks/java/{core/src/test/java/org/apache/beam/sdk/options/RemoteEnvironmentOptionsTest.java
=>
io/fileschematransform/src/test/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProvidersTest.java}
(57%)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new a068e5b0ed5 Updating config from bot
a068e5b0ed5 is described below
commit a068e5b0ed56cd8b80d14dfa0aee690f0f22e4bc
Author: github-actions
AuthorDate: Mon Dec 5 20:10:12 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24532.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24532.json
b/scripts/ci/pr-bot/state/pr-state/pr-24532.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24532.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch release-2.44.0 updated (9eaac2cbc11 -> a7b5818045b)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch release-2.44.0 in repository https://gitbox.apache.org/repos/asf/beam.git from 9eaac2cbc11 Set Dataflow container to release version. add f62381edf96 Update jackson dep. new a7b5818045b Merge pull request #24476: [Cherry Pick #24445] Update jackson dep The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- sdks/java/container/license_scripts/dep_urls_java.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[beam] 01/01: Merge pull request #24476: [Cherry Pick #24445] Update jackson dep
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch release-2.44.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit a7b5818045b5cc04c32342143190daf799f108e0 Merge: 9eaac2cbc11 f62381edf96 Author: Kenneth Knowles AuthorDate: Mon Dec 5 12:08:34 2022 -0800 Merge pull request #24476: [Cherry Pick #24445] Update jackson dep buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- sdks/java/container/license_scripts/dep_urls_java.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 5419bfd78af Updating config from bot 5419bfd78af is described below commit 5419bfd78afc9e9f7944e702bf55467f20c079f8 Author: github-actions AuthorDate: Mon Dec 5 19:33:54 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index f20de24f22e..f80c1766766 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -8,6 +8,7 @@ "tvalentyn": 1669983222906, "pabloem": 1669983210403, "y1chi": 1667002607045, -"damccorm": 1670261644639 +"damccorm": 1670261644639, +"jrmccluskey": 1670268830062 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new a5fb8d747f4 Updating config from bot
a5fb8d747f4 is described below
commit a5fb8d747f482f0ee22e207d118c1751bd11244e
Author: github-actions
AuthorDate: Mon Dec 5 19:33:51 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24530.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24530.json
b/scripts/ci/pr-bot/state/pr-state/pr-24530.json
new file mode 100644
index 000..959c289c017
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24530.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "jrmccluskey"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: Bump cloud.google.com/go/bigtable from 1.18.0 to 1.18.1 in /sdks (#24500)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 9f2de2ab8ed Bump cloud.google.com/go/bigtable from 1.18.0 to 1.18.1 in /sdks (#24500) 9f2de2ab8ed is described below commit 9f2de2ab8ed4dda2411f97496515fb2f2cd9e22d Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Dec 5 14:00:36 2022 -0500 Bump cloud.google.com/go/bigtable from 1.18.0 to 1.18.1 in /sdks (#24500) Bumps [cloud.google.com/go/bigtable](https://github.com/googleapis/google-cloud-go) from 1.18.0 to 1.18.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.18.0...storage/v1.18.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigtable dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index cb72104daac..1b01fcda421 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -64,7 +64,7 @@ require ( require cloud.google.com/go/spanner v1.41.0 require ( - cloud.google.com/go/bigtable v1.18.0 + cloud.google.com/go/bigtable v1.18.1 github.com/tetratelabs/wazero v1.0.0-pre.4 ) diff --git a/sdks/go.sum b/sdks/go.sum index d9561e0a05a..c7dcb6a2b5a 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -18,8 +18,8 @@ cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNF cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc= cloud.google.com/go/bigquery v1.44.0 h1:Wi4dITi+cf9VYp4VH2T9O41w0kCW0uQTELq2Z6tukN0= cloud.google.com/go/bigquery v1.44.0/go.mod h1:0Y33VqXTEsbamHJvJHdFmtqHvMIY28aK1+dFsvaChGc= -cloud.google.com/go/bigtable v1.18.0 h1:OzxQqEBRNcUt0u3V9HobUS95hr1GVVPNHtPGrCeXBfU= -cloud.google.com/go/bigtable v1.18.0/go.mod h1:TwTdxeNeIwj2lOmtvqISXlRWuIovWkjSZsd03sCLz2U= +cloud.google.com/go/bigtable v1.18.1 h1:SxQk9Bj6OKxeiuvevG/KBjqGn/7X8heZbWfK0tYkFd8= +cloud.google.com/go/bigtable v1.18.1/go.mod h1:NAVyfJot9jlo+KmgWLUJ5DJGwNDoChzAcrecLpmuAmY= cloud.google.com/go/compute v1.12.1 h1:gKVJMEyqV5c/UnpzjjQbo3Rjvvqpr9B1DFSbJC4OXr0= cloud.google.com/go/compute v1.12.1/go.mod h1:e8yNOBcBONZU1vJKCvCoDw/4JQsA0dpM4x/6PIIOocU= cloud.google.com/go/compute/metadata v0.2.1 h1:efOwf5ymceDhK6PKMnnrTHP4pppY5L22mle96M1yP48=
[beam] branch master updated: Bump golang.org/x/sys from 0.2.0 to 0.3.0 in /sdks (#24519)
This is an automated email from the ASF dual-hosted git repository. jrmccluskey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 6ba427ea6a1 Bump golang.org/x/sys from 0.2.0 to 0.3.0 in /sdks (#24519) 6ba427ea6a1 is described below commit 6ba427ea6a1e1ae02cc388d90543a3ecb448ef2c Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Dec 5 14:00:19 2022 -0500 Bump golang.org/x/sys from 0.2.0 to 0.3.0 in /sdks (#24519) Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.2.0 to 0.3.0. - [Release notes](https://github.com/golang/sys/releases) - [Commits](https://github.com/golang/sys/compare/v0.2.0...v0.3.0) --- updated-dependencies: - dependency-name: golang.org/x/sys dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index 66e86d3866f..cb72104daac 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -51,7 +51,7 @@ require ( golang.org/x/net v0.2.0 golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 golang.org/x/sync v0.1.0 - golang.org/x/sys v0.2.0 + golang.org/x/sys v0.3.0 golang.org/x/text v0.5.0 google.golang.org/api v0.103.0 google.golang.org/genproto v0.0.0-20221201164419-0e50fba7f41c diff --git a/sdks/go.sum b/sdks/go.sum index 2775aef863e..d9561e0a05a 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -1050,8 +1050,8 @@ golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-2026061358-0a5406a5449c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.2.0 h1:ljd4t30dBnAvMZaQCevtY0xLLD0A+bRZXbgLMLU1F/A= -golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.3.0 h1:w8ZOecv6NaNa/zC8944JTU3vz4u6Lagfk4RPQxv92NQ= +golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
[beam] 01/01: Update dataflow container version for Pandas upgrade
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch users/damccorm/pandasContainer in repository https://gitbox.apache.org/repos/asf/beam.git commit 2af77a8fc00eaf2702ffd3450725e7fc437df90c Author: Danny McCormick AuthorDate: Mon Dec 5 13:56:55 2022 -0500 Update dataflow container version for Pandas upgrade --- sdks/python/apache_beam/runners/dataflow/internal/names.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py b/sdks/python/apache_beam/runners/dataflow/internal/names.py index 34ef156415c..9bd9bf55057 100644 --- a/sdks/python/apache_beam/runners/dataflow/internal/names.py +++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py @@ -36,10 +36,10 @@ SERIALIZED_SOURCE_KEY = 'serialized_source' # Update this version to the next version whenever there is a change that will # require changes to legacy Dataflow worker execution environment. -BEAM_CONTAINER_VERSION = 'beam-master-20221122' +BEAM_CONTAINER_VERSION = 'beam-master-20221205' # Update this version to the next version whenever there is a change that # requires changes to SDK harness container or SDK harness launcher. -BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20221122' +BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20221205' DATAFLOW_CONTAINER_IMAGE_REPOSITORY = 'gcr.io/cloud-dataflow/v1beta3'
[beam] branch users/damccorm/pandasContainer created (now 2af77a8fc00)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/pandasContainer in repository https://gitbox.apache.org/repos/asf/beam.git at 2af77a8fc00 Update dataflow container version for Pandas upgrade This branch includes the following new commits: new 2af77a8fc00 Update dataflow container version for Pandas upgrade The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 28cbcd79fbc Updating config from bot
28cbcd79fbc is described below
commit 28cbcd79fbc011a32d95baca2dcfd949dda3f627
Author: github-actions
AuthorDate: Mon Dec 5 18:52:21 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24529.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24529.json
b/scripts/ci/pr-bot/state/pr-state/pr-24529.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24529.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: Cloudbuild+manualsetup+playground (#24144)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 9b3b403ae9c Cloudbuild+manualsetup+playground (#24144) 9b3b403ae9c is described below commit 9b3b403ae9c071dcef0e801c892eab2203db88df Author: ruslan-ikhsan <114978215+ruslan-ikh...@users.noreply.github.com> AuthorDate: Mon Dec 5 23:46:30 2022 +0500 Cloudbuild+manualsetup+playground (#24144) * Commit related to apache #24007, #24006 * Commit related to apache #24007, #24006 * Update README.md * Updates related to apache #24007, #24006 * Updates to comments in apache #24144 * Update README.md * Update README.md * Update README.md * Optimizing cloud build yaml files * Updates related to apache #24007, #24006 * Updates related to apache #24006, #24007 * Update README.md * Update cloudbuild_pg_to_gke.yaml * Added curly brackets for variables * Update cloudbuild_pg_to_gke.yaml * Test updates for apache #24007 * Create terraform.tfvars * Latest updates related to PR apache #24144 * Delete terraform.tfvars * Update README.md * Update README.md * Update README.md * Updates related to #24144 * Update .gitignore * Updates related to PR #24144 * Update README.md * Updates for apache #24144 * Update README.md * Update README.md * Update variables.tf * Update state.tfbackend * Revert "Update state.tfbackend" This reverts commit 0aa6f1ffd81854e950b35ae5b36337e0eb693372. * Update README.md * Update README.md * Update README.md * Update variables and logging type #24144 * Updates related to tfvars file and Readme * Updates related to apache beam PR #24144 * Updates related to apache beam PR #24144 * Update iam.tf * Update cloudbuild_pg_infra.yaml * Updating cloud build configs * Update provider version to 4.0 * Update README.md * Updates with bash scripts for cloud builder * Update cloudbuild_pg_infra.yaml * Update cloudbuild_pg_infra.yaml * Updates of cloud build logic * Update env_init.sh * Update cloudbuild_pg_infra.yaml * Update cloudbuild_pg_infra.yaml * Updates to cb bash scripts * Update cloudbuild_pg_infra.yaml * Update cloudbuild_pg_infra.yaml * Shell scripts updates * Update cloudbuild_pg_infra.yaml * Update cloudbuild_pg_infra.yaml * Shell script updates cloud build * Updates * Updates related to PR #24144 * Update terraform.tf * Update README.md * Updates related to PR #24144 * Update cloudbuild_pg_to_gke.yaml * Revert "Update cloudbuild_pg_to_gke.yaml" This reverts commit 6545170605995fcf75cb628dee1c640ecd90f864. * Update triggers.tf * Updates with new variables for region/location #24144 * Update README.md * Updated for SDK_TAG * Update README.md * Update cloudbuild_pg_to_gke.yaml * Update cloudbuild_pg_to_gke.yaml * Update README.md * Update build.gradle.kts * Updates * Update build.gradle.kts * Update cloudbuild_pg_to_gke.yaml * Update Dockerfile * Update cloudbuild_pg_to_gke.yaml * Update build.gradle.kts * test faster without back * Update env_init.sh * Updates to cloud build * Update cloudbuild_pg_to_gke.yaml * Update build.gradle.kts * Updates cloud build * Update cloudbuild_pg_to_gke.yaml * Update env_init.sh * Updates * Update build.gradle.kts * Updates * trying to set run helm verbose * remove unneeded task temporarily * remove extraargs * verbose * remove extra args * remove commented code * Update build.gradle.kts * Update build.gradle.kts * Update build.gradle.kts * Update build.gradle.kts * Update build.gradle.kts * Update cloudbuild_pg_to_gke.yaml * Update cloudbuild_pg_to_gke.yaml * Update build.gradle.kts * Update build.gradle.kts * Update build.gradle.kts * Updates related to PR #24144 * Updates related to PR #24144 * Update README.md * Update build.gradle.kts * Update README.md * Update README.md * Update README.md * Update variables.tf * Update main.tf * Revert "Update main.tf" Th
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 4cb48900876 Updating config from bot 4cb48900876 is described below commit 4cb489008767bb25559f28a8f0771bf4c211bae7 Author: github-actions AuthorDate: Mon Dec 5 18:44:43 2022 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-24495.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24495.json b/scripts/ci/pr-bot/state/pr-state/pr-24495.json index eb0c7805a62..438c2562a35 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-24495.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-24495.json @@ -5,7 +5,7 @@ "io": "johnjcasey" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": false } \ No newline at end of file
[beam] branch master updated: Run go fmt over full go directory with go 1.19 (#24525)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 33af390d3a4 Run go fmt over full go directory with go 1.19 (#24525) 33af390d3a4 is described below commit 33af390d3a4571893b1865d4a816bd8e4aaa2e95 Author: Danny McCormick AuthorDate: Mon Dec 5 13:29:53 2022 -0500 Run go fmt over full go directory with go 1.19 (#24525) --- sdks/go/cmd/starcgen/starcgen.go | 15 +-- sdks/go/examples/cookbook/tornadoes/tornadoes.go | 4 +- .../debugging_wordcount/debugging_wordcount.go | 8 +- sdks/go/examples/fhirio/import/import.go | 33 ++--- .../fhirio/read_write_pubsub/read_write_pubsub.go | 37 ++--- sdks/go/examples/kafka/taxi.go | 62 + .../go/examples/large_wordcount/large_wordcount.go | 8 +- .../minimal_wordcount/minimal_wordcount.go | 8 +- sdks/go/examples/native_wordcap/wordcap.go | 7 +- sdks/go/examples/streaming_wordcap/wordcap.go | 7 +- sdks/go/examples/stringsplit/stringsplit.go| 6 +- sdks/go/examples/xlang/bigquery/wordcount.go | 61 + sdks/go/examples/yatzy/yatzy.go| 2 +- sdks/go/pkg/beam/core/funcx/fn.go | 6 +- sdks/go/pkg/beam/core/funcx/output.go | 7 +- sdks/go/pkg/beam/core/funcx/sideinput.go | 4 +- sdks/go/pkg/beam/core/funcx/signature.go | 6 +- sdks/go/pkg/beam/core/graph/bind.go| 18 +-- sdks/go/pkg/beam/core/graph/coder/registry.go | 8 +- sdks/go/pkg/beam/core/graph/fn.go | 19 +-- .../beam/core/graph/window/trigger/trigger_test.go | 2 +- sdks/go/pkg/beam/core/metrics/metrics.go | 7 +- sdks/go/pkg/beam/core/runtime/exec/combine_test.go | 32 ++--- sdks/go/pkg/beam/core/runtime/exec/fn_test.go | 10 +- sdks/go/pkg/beam/core/runtime/exec/sdf.go | 150 +++-- .../beam/core/runtime/exec/sdf_invokers_test.go| 1 - sdks/go/pkg/beam/core/runtime/xlangx/registry.go | 11 +- sdks/go/pkg/beam/core/runtime/xlangx/resolve.go| 9 +- sdks/go/pkg/beam/core/util/ioutilx/read.go | 1 + sdks/go/pkg/beam/forward.go| 32 ++--- sdks/go/pkg/beam/gbk.go| 4 +- sdks/go/pkg/beam/internal/errors/errors.go | 12 +- sdks/go/pkg/beam/io/databaseio/mapper.go | 10 +- sdks/go/pkg/beam/io/databaseio/util.go | 2 +- sdks/go/pkg/beam/io/datastoreio/datastore.go | 1 - sdks/go/pkg/beam/io/parquetio/parquetio.go | 38 +++--- sdks/go/pkg/beam/io/synthetic/source.go| 27 ++-- sdks/go/pkg/beam/io/synthetic/step.go | 6 +- sdks/go/pkg/beam/io/xlang/bigqueryio/bigquery.go | 71 +- sdks/go/pkg/beam/io/xlang/debeziumio/debezium.go | 32 ++--- sdks/go/pkg/beam/io/xlang/jdbcio/jdbc.go | 71 +- sdks/go/pkg/beam/io/xlang/kafkaio/kafka.go | 31 ++--- .../beam/model/fnexecution_v1/beam_fn_api.pb.go| 38 +++--- .../beam/model/jobmanagement_v1/beam_job_api.pb.go | 11 +- .../beam/model/pipeline_v1/beam_runner_api.pb.go | 42 +++--- sdks/go/pkg/beam/pardo.go | 146 ++-- sdks/go/pkg/beam/partition.go | 4 +- .../beam/runners/dataflow/dataflowlib/messages.go | 18 +-- .../beam/runners/universal/runnerlib/compile.go| 8 +- sdks/go/pkg/beam/schema.go | 10 +- sdks/go/pkg/beam/testing/ptest/ptest.go| 1 - sdks/go/pkg/beam/transforms/filter/filter.go | 16 +-- sdks/go/pkg/beam/transforms/stats/max.go | 5 +- sdks/go/pkg/beam/transforms/stats/mean.go | 5 +- sdks/go/pkg/beam/transforms/stats/min.go | 5 +- sdks/go/pkg/beam/transforms/stats/sum.go | 5 +- sdks/go/pkg/beam/transforms/top/top.go | 10 +- .../pkg/beam/transforms/xlang/python/external.go | 31 +++-- sdks/go/pkg/beam/xlang.go | 61 + sdks/go/test/integration/expansions.go | 23 ++-- sdks/go/test/integration/flags.go | 9 +- .../go/test/integration/io/xlang/jdbc/jdbc_test.go | 2 +- 62 files changed, 693 insertions(+), 643 deletions(-) diff --git a/sdks/go/cmd/starcgen/starcgen.go b/sdks/go/cmd/starcgen/starcgen.go index d6e6f1f10c5..9e9859b202b 100644 --- a/sdks/go/cmd/starcgen/starcgen.go +++ b/sdks/go/cmd/starcgen/starcgen.go @@ -21,15 +21,15 @@ // reflection based shims used by beam. Reflection is convenient for development, // but is an unnecessary expense on pipeline performance. // -// Using This Tool +// # Using This Tool // // This tool is intended for use with `go generate`. The recommended convention // putting the types and f
[beam] branch master updated: Support for JsonSchema in Kafka Read Schema Transform (#24272)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new dbb584997c7 Support for JsonSchema in Kafka Read Schema Transform
(#24272)
dbb584997c7 is described below
commit dbb584997c74766c10f8d9ad15d960bedf82e6f2
Author: Pablo Estrada
AuthorDate: Mon Dec 5 10:22:28 2022 -0800
Support for JsonSchema in Kafka Read Schema Transform (#24272)
* Implement KafkaReadSchemaTransform support for JSON
* fixup
* add dependencies
* fixup
---
sdks/java/io/kafka/build.gradle| 3 ++
... => KafkaReadSchemaTransformConfiguration.java} | 16 +++
java => KafkaReadSchemaTransformProvider.java} | 30 +++-
...a => KafkaReadSchemaTransformProviderTest.java} | 55 +-
4 files changed, 72 insertions(+), 32 deletions(-)
diff --git a/sdks/java/io/kafka/build.gradle b/sdks/java/io/kafka/build.gradle
index 68ce449394b..5e3680a74ea 100644
--- a/sdks/java/io/kafka/build.gradle
+++ b/sdks/java/io/kafka/build.gradle
@@ -85,6 +85,9 @@ dependencies {
// "kafka-clients" has to be provided since user can use its own version.
exclude group: "org.apache.kafka", module: "kafka-clients"
}
+ // everit_json is needed for Kafka Read SchemaTransform tests that rely on
JSON-schema translation.
+ permitUnusedDeclared library.java.everit_json_schema
+ provided library.java.everit_json_schema
testImplementation project(path: ":sdks:java:core", configuration:
"shadowTest")
testImplementation project(":sdks:java:io:synthetic")
testImplementation project(path: ":sdks:java:io:common", configuration:
"testRuntimeMigration")
diff --git
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaSchemaTransformReadConfiguration.java
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformConfiguration.java
similarity index 88%
rename from
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaSchemaTransformReadConfiguration.java
rename to
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformConfiguration.java
index 9280de6c13f..b930e7baa46 100644
---
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaSchemaTransformReadConfiguration.java
+++
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformConfiguration.java
@@ -36,7 +36,7 @@ import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
@Experimental
@DefaultSchema(AutoValueSchema.class)
@AutoValue
-public abstract class KafkaSchemaTransformReadConfiguration {
+public abstract class KafkaReadSchemaTransformConfiguration {
public static final Set VALID_START_OFFSET_VALUES =
Sets.newHashSet("earliest", "latest");
public static final Set VALID_DATA_FORMATS = Sets.newHashSet("AVRO",
"JSON");
@@ -50,9 +50,9 @@ public abstract class KafkaSchemaTransformReadConfiguration {
: "Valid data formats are " + VALID_DATA_FORMATS;
}
- /** Instantiates a {@link KafkaSchemaTransformReadConfiguration.Builder}
instance. */
+ /** Instantiates a {@link KafkaReadSchemaTransformConfiguration.Builder}
instance. */
public static Builder builder() {
-return new AutoValue_KafkaSchemaTransformReadConfiguration.Builder();
+return new AutoValue_KafkaReadSchemaTransformConfiguration.Builder();
}
/** Sets the bootstrap servers for the Kafka consumer. */
@@ -69,7 +69,7 @@ public abstract class KafkaSchemaTransformReadConfiguration {
public abstract String getConfluentSchemaRegistrySubject();
@Nullable
- public abstract String getAvroSchema();
+ public abstract String getSchema();
@Nullable
public abstract String getAutoOffsetResetConfig();
@@ -80,7 +80,7 @@ public abstract class KafkaSchemaTransformReadConfiguration {
/** Sets the topic from which to read. */
public abstract String getTopic();
- /** Builder for the {@link KafkaSchemaTransformReadConfiguration}. */
+ /** Builder for the {@link KafkaReadSchemaTransformConfiguration}. */
@AutoValue.Builder
public abstract static class Builder {
@@ -91,7 +91,7 @@ public abstract class KafkaSchemaTransformReadConfiguration {
public abstract Builder setConfluentSchemaRegistrySubject(String subject);
-public abstract Builder setAvroSchema(String schema);
+public abstract Builder setSchema(String schema);
public abstract Builder setDataFormat(String dataFormat);
@@ -102,7 +102,7 @@ public abstract class KafkaSchemaTransformReadConfiguration
{
/** Sets the topic from which to read. */
public abstract Builder setTopic(String value);
-/** Builds a {@link KafkaSchemaTransformReadConfiguration} instance. */
-public abstract KafkaSchemaTransformReadConfiguration build();
+/** Builds a {@link Kaf
[beam] branch master updated (7701db80691 -> 3e4051cd005)
This is an automated email from the ASF dual-hosted git repository. ningk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 7701db80691 Bump golang.org/x/text from 0.4.0 to 0.5.0 in /sdks (#24520) add 7e6e13e5864 Update nbconvert requirement in /sdks/python add 3e4051cd005 Merge pull request #23240 from apache/dependabot/pip/sdks/python/nbconvert-gte-6.2.0-and-lt-8 No new revisions were added by this update. Summary of changes: sdks/python/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new adab2a5dabe Updating config from bot adab2a5dabe is described below commit adab2a5dabe672bb01e7886136d69c47d7a53cd6 Author: github-actions AuthorDate: Mon Dec 5 17:34:08 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 07770f625de..f20de24f22e 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -7,6 +7,7 @@ "ryanthompson591": 1670002443548, "tvalentyn": 1669983222906, "pabloem": 1669983210403, -"y1chi": 1667002607045 +"y1chi": 1667002607045, +"damccorm": 1670261644639 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 99edecb84ca Updating config from bot
99edecb84ca is described below
commit 99edecb84ca4357a9cc7b569d41845b77a839c2e
Author: github-actions
AuthorDate: Mon Dec 5 17:34:06 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24526.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24526.json
b/scripts/ci/pr-bot/state/pr-state/pr-24526.json
new file mode 100644
index 000..88d1920a8bb
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24526.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch dependabot/go_modules/sdks/golang.org/x/sys-0.3.0 updated (aa96d6d6500 -> 8d03012d0ac)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/golang.org/x/sys-0.3.0 in repository https://gitbox.apache.org/repos/asf/beam.git discard aa96d6d6500 Bump golang.org/x/sys from 0.2.0 to 0.3.0 in /sdks add a9dcf95453d [CdapIO] Add readme for CdapIO. Update readme for SparkReceiverIO. (#23959) add c197579e9f2 [Spark Dataset runner] Add @Experimental and reduce visibility where possible. add 66db2d81811 Merge pull request #24522: [Spark Dataset runner] Add @Experimental and reduce visibility where possible add 54f579e45ab Fix grafana dashboard id (#24524) add dd8c035937a [Spark runner] Support running (VR) tests with Java 17 (closes #24400) (#24401) add 12d0ba99d19 Replaced deprecated finalize with DoFn Teardown (#24516) add 719e5946420 Bump cloud.google.com/go/storage from 1.28.0 to 1.28.1 in /sdks (#24517) add a88956b1e7c add clarifier to error message (#24449) add 4a45066e653 Batch rename requests in fileio.WriteToFiles (#24341) add 7701db80691 Bump golang.org/x/text from 0.4.0 to 0.5.0 in /sdks (#24520) add 8d03012d0ac Bump golang.org/x/sys from 0.2.0 to 0.3.0 in /sdks This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (aa96d6d6500) \ N -- N -- N refs/heads/dependabot/go_modules/sdks/golang.org/x/sys-0.3.0 (8d03012d0ac) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../Python_ML_RunInference_Benchmark_Tests.json| 2 +- .../SparkStructuredStreamingPipelineOptions.java | 2 + .../SparkStructuredStreamingRunner.java| 2 + .../translation/batch/Aggregators.java | 8 +- .../batch/CombineGloballyTranslatorBatch.java | 2 +- .../batch/CombineGroupedValuesTranslatorBatch.java | 2 +- .../batch/GroupByKeyTranslatorBatch.java | 4 +- .../translation/batch/ImpulseTranslatorBatch.java | 3 +- .../batch/ReshuffleTranslatorBatch.java| 4 +- runners/spark/spark_runner.gradle | 30 - sdks/go.mod| 4 +- sdks/go.sum| 8 +- sdks/java/io/cdap/README.md| 145 + sdks/java/io/cdap/build.gradle | 1 - .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 4 +- sdks/java/io/sparkreceiver/2/README.md | 40 +- sdks/java/io/sparkreceiver/2/build.gradle | 1 - sdks/python/apache_beam/io/filebasedsink.py| 3 +- sdks/python/apache_beam/io/fileio.py | 65 + sdks/python/apache_beam/io/gcp/gcsio.py| 5 +- 20 files changed, 274 insertions(+), 61 deletions(-) create mode 100644 sdks/java/io/cdap/README.md
[beam] branch master updated: Bump golang.org/x/text from 0.4.0 to 0.5.0 in /sdks (#24520)
This is an automated email from the ASF dual-hosted git repository. riteshghorse pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 7701db80691 Bump golang.org/x/text from 0.4.0 to 0.5.0 in /sdks (#24520) 7701db80691 is described below commit 7701db8069168d58db582195feee6162cea4865e Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Dec 5 23:02:11 2022 +0530 Bump golang.org/x/text from 0.4.0 to 0.5.0 in /sdks (#24520) Bumps [golang.org/x/text](https://github.com/golang/text) from 0.4.0 to 0.5.0. - [Release notes](https://github.com/golang/text/releases) - [Commits](https://github.com/golang/text/compare/v0.4.0...v0.5.0) --- updated-dependencies: - dependency-name: golang.org/x/text dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index b88fb54fc1b..66e86d3866f 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -52,7 +52,7 @@ require ( golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 golang.org/x/sync v0.1.0 golang.org/x/sys v0.2.0 - golang.org/x/text v0.4.0 + golang.org/x/text v0.5.0 google.golang.org/api v0.103.0 google.golang.org/genproto v0.0.0-20221201164419-0e50fba7f41c google.golang.org/grpc v1.51.0 diff --git a/sdks/go.sum b/sdks/go.sum index 98faee6ea89..2775aef863e 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -1060,8 +1060,8 @@ golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.4.0 h1:BrVqGRd7+k1DiOgtnFvAkoQEWQvBc25ouMJM6429SFg= -golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.5.0 h1:OLmvp0KP+FVG99Ct/qFiL/Fhk4zp4QQnZ7b2U+5piUM= +golang.org/x/text v0.5.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
[beam] branch master updated (a88956b1e7c -> 4a45066e653)
This is an automated email from the ASF dual-hosted git repository. johncasey pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a88956b1e7c add clarifier to error message (#24449) add 4a45066e653 Batch rename requests in fileio.WriteToFiles (#24341) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/filebasedsink.py | 3 +- sdks/python/apache_beam/io/fileio.py| 65 + 2 files changed, 39 insertions(+), 29 deletions(-)
[beam] branch master updated: add clarifier to error message (#24449)
This is an automated email from the ASF dual-hosted git repository. johncasey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new a88956b1e7c add clarifier to error message (#24449) a88956b1e7c is described below commit a88956b1e7c735f9f2ed61764024856d88bd64c3 Author: Andrei Gurau AuthorDate: Mon Dec 5 12:18:35 2022 -0500 add clarifier to error message (#24449) remove file --- sdks/python/apache_beam/io/gcp/gcsio.py | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/io/gcp/gcsio.py b/sdks/python/apache_beam/io/gcp/gcsio.py index 5aee4f5aaf3..2010d229259 100644 --- a/sdks/python/apache_beam/io/gcp/gcsio.py +++ b/sdks/python/apache_beam/io/gcp/gcsio.py @@ -823,4 +823,7 @@ class GcsUploader(Uploader): self._upload_thread.join() # Check for exception since the last put() call. if self._upload_thread.last_error is not None: - raise self._upload_thread.last_error # pylint: disable=raising-bad-type + raise type(self._upload_thread.last_error)( + "Error while uploading file %s: %s", + self._path, + self._upload_thread.last_error.message) # pylint: disable=raising-bad-type
[beam] branch dependabot/go_modules/sdks/cloud.google.com/go/pubsub-1.27.1 updated (763081146e5 -> 5c150e9f52d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/pubsub-1.27.1 in repository https://gitbox.apache.org/repos/asf/beam.git discard 763081146e5 Bump cloud.google.com/go/pubsub from 1.27.0 to 1.27.1 in /sdks add a9dcf95453d [CdapIO] Add readme for CdapIO. Update readme for SparkReceiverIO. (#23959) add c197579e9f2 [Spark Dataset runner] Add @Experimental and reduce visibility where possible. add 66db2d81811 Merge pull request #24522: [Spark Dataset runner] Add @Experimental and reduce visibility where possible add 54f579e45ab Fix grafana dashboard id (#24524) add dd8c035937a [Spark runner] Support running (VR) tests with Java 17 (closes #24400) (#24401) add 12d0ba99d19 Replaced deprecated finalize with DoFn Teardown (#24516) add 719e5946420 Bump cloud.google.com/go/storage from 1.28.0 to 1.28.1 in /sdks (#24517) add 5c150e9f52d Bump cloud.google.com/go/pubsub from 1.27.0 to 1.27.1 in /sdks This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (763081146e5) \ N -- N -- N refs/heads/dependabot/go_modules/sdks/cloud.google.com/go/pubsub-1.27.1 (5c150e9f52d) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../Python_ML_RunInference_Benchmark_Tests.json| 2 +- .../SparkStructuredStreamingPipelineOptions.java | 2 + .../SparkStructuredStreamingRunner.java| 2 + .../translation/batch/Aggregators.java | 8 +- .../batch/CombineGloballyTranslatorBatch.java | 2 +- .../batch/CombineGroupedValuesTranslatorBatch.java | 2 +- .../batch/GroupByKeyTranslatorBatch.java | 4 +- .../translation/batch/ImpulseTranslatorBatch.java | 3 +- .../batch/ReshuffleTranslatorBatch.java| 4 +- runners/spark/spark_runner.gradle | 30 - sdks/go.mod| 2 +- sdks/go.sum| 4 +- sdks/java/io/cdap/README.md| 145 + sdks/java/io/cdap/build.gradle | 1 - .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 4 +- sdks/java/io/sparkreceiver/2/README.md | 40 +- sdks/java/io/sparkreceiver/2/build.gradle | 1 - 17 files changed, 228 insertions(+), 28 deletions(-) create mode 100644 sdks/java/io/cdap/README.md
[beam] branch master updated: Bump cloud.google.com/go/storage from 1.28.0 to 1.28.1 in /sdks (#24517)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 719e5946420 Bump cloud.google.com/go/storage from 1.28.0 to 1.28.1 in /sdks (#24517) 719e5946420 is described below commit 719e59464207a1b686c90d2c34b68ec572c3edf8 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Dec 5 11:26:23 2022 -0500 Bump cloud.google.com/go/storage from 1.28.0 to 1.28.1 in /sdks (#24517) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.28.0 to 1.28.1. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.28.0...storage/v1.28.1) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- sdks/go.mod | 2 +- sdks/go.sum | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/go.mod b/sdks/go.mod index ead567846fd..b88fb54fc1b 100644 --- a/sdks/go.mod +++ b/sdks/go.mod @@ -27,7 +27,7 @@ require ( cloud.google.com/go/datastore v1.10.0 cloud.google.com/go/profiler v0.3.1 cloud.google.com/go/pubsub v1.27.0 - cloud.google.com/go/storage v1.28.0 + cloud.google.com/go/storage v1.28.1 github.com/aws/aws-sdk-go-v2 v1.17.2 github.com/aws/aws-sdk-go-v2/config v1.18.4 github.com/aws/aws-sdk-go-v2/credentials v1.13.4 diff --git a/sdks/go.sum b/sdks/go.sum index 9487fc6af9b..98faee6ea89 100644 --- a/sdks/go.sum +++ b/sdks/go.sum @@ -46,8 +46,8 @@ cloud.google.com/go/spanner v1.41.0/go.mod h1:MLYDBJR/dY4Wt7ZaMIQ7rXOTLjYrmxLE/5 cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos= cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk= -cloud.google.com/go/storage v1.28.0 h1:DLrIZ6xkeZX6K70fU/boWx5INJumt6f+nwwWSHXzzGY= -cloud.google.com/go/storage v1.28.0/go.mod h1:qlgZML35PXA3zoEnIkiPLY4/TOkUleufRlu6qmcf7sI= +cloud.google.com/go/storage v1.28.1 h1:F5QDG5ChchaAVQhINh24U99OWHURqrW8OmQcGKXcbgI= +cloud.google.com/go/storage v1.28.1/go.mod h1:Qnisd4CqDdo6BGs2AD5LLnEsmSQ80wQ5ogcBBKhU86Y= dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= github.com/Azure/azure-pipeline-go v0.2.3/go.mod h1:x841ezTBIMG6O3lAcl8ATHnsOPVl2bqk7S3ta6S6u4k= github.com/Azure/azure-sdk-for-go v16.2.1+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
[beam] branch master updated: Replaced deprecated finalize with DoFn Teardown (#24516)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 12d0ba99d19 Replaced deprecated finalize with DoFn Teardown (#24516)
12d0ba99d19 is described below
commit 12d0ba99d194d0dc3f426886f91f420cd8d337af
Author: Dejan Spasic
AuthorDate: Mon Dec 5 16:35:57 2022 +0100
Replaced deprecated finalize with DoFn Teardown (#24516)
Since the deprecation of the finalize() method in Java 9, it is
recommended replacing the method with the DoFn.Teardown
annotation.
Performed the renaming of the method to tearDown and increasing the
visibility to public.
[Task: #24181]
---
.../io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
index 97fd2a41ed1..1c103989972 100644
--- a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
@@ -1409,8 +1409,8 @@ public class JdbcIO {
cleanUpConnection();
}
-@Override
-protected void finalize() throws Throwable {
+@Teardown
+public void tearDown() throws Exception {
cleanUpConnection();
}
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 3a4ef951753 Updating config from bot 3a4ef951753 is described below commit 3a4ef951753a567676f44acc58e0090b6812b549 Author: github-actions AuthorDate: Mon Dec 5 15:27:08 2022 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-24074.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24074.json b/scripts/ci/pr-bot/state/pr-state/pr-24074.json index f0f36de0e32..59981e12408 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-24074.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-24074.json @@ -4,7 +4,7 @@ "java": "kennknowles", "io": "Abacn" }, - "nextAction": "Reviewers", + "nextAction": "Author", "stopReviewerNotifications": false, "remindAfterTestsPass": [], "committerAssigned": false
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 4f9c3444b7a Updating config from bot
4f9c3444b7a is described below
commit 4f9c3444b7a864c0c3e438cc94dfca93ecc7ac24
Author: github-actions
AuthorDate: Mon Dec 5 15:13:05 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24525.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24525.json
b/scripts/ci/pr-bot/state/pr-state/pr-24525.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24525.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated: [Spark runner] Support running (VR) tests with Java 17 (closes #24400) (#24401)
This is an automated email from the ASF dual-hosted git repository.
echauchot pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new dd8c035937a [Spark runner] Support running (VR) tests with Java 17
(closes #24400) (#24401)
dd8c035937a is described below
commit dd8c035937a275246a7901718d7bcd890536731b
Author: Moritz Mack
AuthorDate: Mon Dec 5 15:58:26 2022 +0100
[Spark runner] Support running (VR) tests with Java 17 (closes #24400)
(#24401)
---
runners/spark/spark_runner.gradle | 30 ++
1 file changed, 26 insertions(+), 4 deletions(-)
diff --git a/runners/spark/spark_runner.gradle
b/runners/spark/spark_runner.gradle
index 365b0935151..1c04aef4d8c 100644
--- a/runners/spark/spark_runner.gradle
+++ b/runners/spark/spark_runner.gradle
@@ -61,6 +61,22 @@ def sparkTestProperties(overrides = [:]) {
]
}
+
+def sparkTestJvmArgs() {
+ // run tests with Java 17 using -PcompileAndRunTestsWithJava17
-Pjava17Home=???
+ if (project.hasProperty("compileAndRunTestsWithJava17")) {
+return [
+ "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
+ // add-opens below required for Kryo FieldSerializer /
SparkRunnerKryoRegistratorTest
+ "--add-opens=java.base/java.nio=ALL-UNNAMED",
+ "--add-opens=java.base/java.util=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED"
+]
+ } else {
+return []
+ }
+}
+
def hadoopVersions = [
"285" : "2.8.5",
"292" : "2.9.2",
@@ -113,6 +129,7 @@ test {
// Change log level to debug only for the package and nested packages:
// systemProperty
"org.slf4j.simpleLogger.log.org.apache.beam.runners.spark.stateful", "debug"
jvmArgs "-XX:-UseGCOverheadLimit"
+ jvmArgs += sparkTestJvmArgs()
if (System.getProperty("beamSurefireArgline")) {
jvmArgs System.getProperty("beamSurefireArgline")
}
@@ -230,6 +247,8 @@ def validatesRunnerBatch =
tasks.register("validatesRunnerBatch", Test) {
// Disable gradle cache
outputs.upToDateWhen { false }
systemProperties sparkTestProperties(["--enableSparkMetricSinks":"false"])
+ jvmArgs += sparkTestJvmArgs()
+ jvmArgs '-Xmx3g'
classpath = configurations.validatesRunner
testClassesDirs = files(
@@ -263,7 +282,6 @@ def validatesRunnerBatch =
tasks.register("validatesRunnerBatch", Test) {
excludeCategories 'org.apache.beam.sdk.testing.UsesPerKeyOrderedDelivery'
excludeCategories 'org.apache.beam.sdk.testing.UsesPerKeyOrderInBundle'
}
- jvmArgs '-Xmx3g'
}
def validatesRunnerStreaming = tasks.register("validatesRunnerStreaming",
Test) {
@@ -271,6 +289,7 @@ def validatesRunnerStreaming =
tasks.register("validatesRunnerStreaming", Test)
// Disable gradle cache
outputs.upToDateWhen { false }
systemProperties sparkTestProperties(["--enableSparkMetricSinks": "true"])
+ jvmArgs += sparkTestJvmArgs()
classpath = configurations.validatesRunner
testClassesDirs += files(
@@ -330,6 +349,8 @@ tasks.register("validatesStructuredStreamingRunnerBatch",
Test) {
systemProperty 'spark.kryo.classesToRegister',
'org.apache.beam.sdk.transforms.ViewTest$NonDeterministicStringCoder,' +
'org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.RegularImmutableList'
+ jvmArgs += sparkTestJvmArgs()
+ jvmArgs '-Xmx7g' // Increase memory heap in order to avoid OOM errors
classpath = configurations.validatesRunner
testClassesDirs = files(
@@ -339,8 +360,7 @@ tasks.register("validatesStructuredStreamingRunnerBatch",
Test) {
testClassesDirs += files(project.sourceSets.test.output.classesDirs)
maxParallelForks 4
- // Increase memory heap in order to avoid OOM errors
- jvmArgs '-Xmx7g'
+
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
// Should be run only in a properly configured SDK harness environment
@@ -408,6 +428,8 @@ tasks.register("examplesIntegrationTest", Test) {
"--tempRoot": "${tempLocation}",
"--project" : "${gcpProject}"
])
+ jvmArgs += sparkTestJvmArgs()
+ jvmArgs '-Xmx3g'
include '**/*IT.class'
maxParallelForks 4
@@ -420,7 +442,6 @@ tasks.register("examplesIntegrationTest", Test) {
excludeTestsMatching
'org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatchDynamicSharding'
}
}
- jvmArgs '-Xmx3g'
}
hadoopVersions.each { kv ->
@@ -429,6 +450,7 @@ hadoopVersions.each { kv ->
description = "Runs Spark tests with Hadoop version $kv.value"
classpath = configurations."hadoopVersion$kv.key" +
sourceSets.test.runtimeClasspath
systemProperties sparkTestProperties()
+jvmArgs += sparkTestJvmArgs()
include "**/*Test.class"
maxParallelForks 4
[beam] branch master updated: Fix grafana dashboard id (#24524)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 54f579e45ab Fix grafana dashboard id (#24524)
54f579e45ab is described below
commit 54f579e45abcaf774ab3ab65c036dad723b52624
Author: Danny McCormick
AuthorDate: Mon Dec 5 09:30:55 2022 -0500
Fix grafana dashboard id (#24524)
---
.../perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json
b/.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json
index 4c6c56d5b19..fd7f33345bb 100644
---
a/.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json
+++
b/.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json
@@ -32,7 +32,7 @@
"x": 0,
"y": 0
},
- "id": 2,
+ "id": 3,
"options": {
"content": "Detailed descriptions of each benchmark can be found at
https://s.apache.org/beam-ml-metrics";,
"mode": "markdown"
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new dc78676e5ab Updating config from bot
dc78676e5ab is described below
commit dc78676e5ab74143e4477c579e8f43392b678357
Author: github-actions
AuthorDate: Mon Dec 5 14:05:26 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24524.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24524.json
b/scripts/ci/pr-bot/state/pr-state/pr-24524.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24524.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch users/damccorm/dashboardPanelId created (now 50f2de8f893)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/dashboardPanelId in repository https://gitbox.apache.org/repos/asf/beam.git at 50f2de8f893 Fix grafana dashboard id This branch includes the following new commits: new 50f2de8f893 Fix grafana dashboard id The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] 01/01: Fix grafana dashboard id
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/dashboardPanelId
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 50f2de8f893b981534938631eb5828196bca1f0b
Author: Danny McCormick
AuthorDate: Mon Dec 5 09:01:51 2022 -0500
Fix grafana dashboard id
---
.../perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json
b/.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json
index 4c6c56d5b19..fd7f33345bb 100644
---
a/.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json
+++
b/.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json
@@ -32,7 +32,7 @@
"x": 0,
"y": 0
},
- "id": 2,
+ "id": 3,
"options": {
"content": "Detailed descriptions of each benchmark can be found at
https://s.apache.org/beam-ml-metrics";,
"mode": "markdown"
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 8560fc6b819 Updating config from bot 8560fc6b819 is described below commit 8560fc6b81935777fe226153e246eacc3b2fb6d6 Author: github-actions AuthorDate: Mon Dec 5 13:35:41 2022 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-24436.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24436.json b/scripts/ci/pr-bot/state/pr-state/pr-24436.json index 2149d7525a8..4e75edb570c 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-24436.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-24436.json @@ -5,7 +5,7 @@ "io": "pabloem" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": false } \ No newline at end of file
[beam] branch master updated (a9dcf95453d -> 66db2d81811)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a9dcf95453d [CdapIO] Add readme for CdapIO. Update readme for SparkReceiverIO. (#23959) add c197579e9f2 [Spark Dataset runner] Add @Experimental and reduce visibility where possible. new 66db2d81811 Merge pull request #24522: [Spark Dataset runner] Add @Experimental and reduce visibility where possible The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../SparkStructuredStreamingPipelineOptions.java | 2 ++ .../spark/structuredstreaming/SparkStructuredStreamingRunner.java | 2 ++ .../spark/structuredstreaming/translation/batch/Aggregators.java | 8 +--- .../translation/batch/CombineGloballyTranslatorBatch.java | 2 +- .../translation/batch/CombineGroupedValuesTranslatorBatch.java| 2 +- .../translation/batch/GroupByKeyTranslatorBatch.java | 4 ++-- .../translation/batch/ImpulseTranslatorBatch.java | 3 +-- .../translation/batch/ReshuffleTranslatorBatch.java | 4 ++-- 8 files changed, 16 insertions(+), 11 deletions(-)
[beam] 01/01: Merge pull request #24522: [Spark Dataset runner] Add @Experimental and reduce visibility where possible
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 66db2d81811dc26e624f2472415d45e879c076c0 Merge: a9dcf95453d c197579e9f2 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Mon Dec 5 14:08:01 2022 +0100 Merge pull request #24522: [Spark Dataset runner] Add @Experimental and reduce visibility where possible .../SparkStructuredStreamingPipelineOptions.java | 2 ++ .../spark/structuredstreaming/SparkStructuredStreamingRunner.java | 2 ++ .../spark/structuredstreaming/translation/batch/Aggregators.java | 8 +--- .../translation/batch/CombineGloballyTranslatorBatch.java | 2 +- .../translation/batch/CombineGroupedValuesTranslatorBatch.java| 2 +- .../translation/batch/GroupByKeyTranslatorBatch.java | 4 ++-- .../translation/batch/ImpulseTranslatorBatch.java | 3 +-- .../translation/batch/ReshuffleTranslatorBatch.java | 4 ++-- 8 files changed, 16 insertions(+), 11 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 6e1e345b8d1 Updating config from bot
6e1e345b8d1 is described below
commit 6e1e345b8d14c7aa1d7b08b09e0513483f43ef54
Author: github-actions
AuthorDate: Mon Dec 5 12:47:07 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24523.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24523.json
b/scripts/ci/pr-bot/state/pr-state/pr-24523.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24523.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 127cbccff7e Updating config from bot
127cbccff7e is described below
commit 127cbccff7eb229cb8a27cd5aa1934bd03db
Author: github-actions
AuthorDate: Mon Dec 5 12:27:43 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24522.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24522.json
b/scripts/ci/pr-bot/state/pr-state/pr-24522.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24522.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new ef76fd04dea Updating config from bot ef76fd04dea is described below commit ef76fd04dea9a7d488251ab6e1653d53c115fb1d Author: github-actions AuthorDate: Mon Dec 5 12:13:49 2022 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json b/scripts/ci/pr-bot/state/reviewers-for-label-io.json index e607b9bfe1d..914c5100414 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json @@ -4,7 +4,7 @@ "chamikaramj": 1670099635969, "johnjcasey": 1670026448158, "pabloem": 1670088828000, -"Abacn": 1670006143213, +"Abacn": 1670242421955, "ahmedabu98": 1670092513730 } } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new dd546c96143 Updating config from bot
dd546c96143 is described below
commit dd546c961436410b5987af6699b92f6b6246e69c
Author: github-actions
AuthorDate: Mon Dec 5 12:13:46 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
index 840b01ae8e6..4949c275904 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
@@ -2,7 +2,7 @@
"label": "java",
"dateOfLastReviewAssignment": {
"lukecwik": 1670241961663,
-"kennknowles": 1670006143213,
+"kennknowles": 1670242420999,
"robertwb": 1670099635969,
"kileys": 1670088828000,
"apilloud": 1670026448157
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2c5d8edae32 Updating config from bot
2c5d8edae32 is described below
commit 2c5d8edae32735c37d646f19034b8f1aa24d6f8a
Author: github-actions
AuthorDate: Mon Dec 5 12:13:44 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24074.json | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24074.json
b/scripts/ci/pr-bot/state/pr-state/pr-24074.json
index 063ce8593a5..f0f36de0e32 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-24074.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24074.json
@@ -1,8 +1,8 @@
{
"commentedAboutFailingChecks": false,
"reviewersAssignedForLabels": {
-"java": "robertwb",
-"io": "chamikaramj"
+"java": "kennknowles",
+"io": "Abacn"
},
"nextAction": "Reviewers",
"stopReviewerNotifications": false,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 400acab184e Updating config from bot
400acab184e is described below
commit 400acab184e66ca5192cd8918edf8a1f1c77fe24
Author: github-actions
AuthorDate: Mon Dec 5 12:06:08 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index 70a0d075816..2d51ea7398d 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1669983229354,
+"damccorm": 1670241961663,
"Abacn": 1670006143213
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 95855d824ca Updating config from bot
95855d824ca is described below
commit 95855d824caff62103f4578a395e6b6ab8440633
Author: github-actions
AuthorDate: Mon Dec 5 12:06:06 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
index 8e0847d468f..840b01ae8e6 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
@@ -1,7 +1,7 @@
{
"label": "java",
"dateOfLastReviewAssignment": {
-"lukecwik": 1669983229236,
+"lukecwik": 1670241961663,
"kennknowles": 1670006143213,
"robertwb": 1670099635969,
"kileys": 1670088828000,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new f7e19167e20 Updating config from bot
f7e19167e20 is described below
commit f7e19167e20a195cd0aca6efbfc0085a97ab7af6
Author: github-actions
AuthorDate: Mon Dec 5 12:06:03 2022 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-24459.json | 9 ++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-24459.json
b/scripts/ci/pr-bot/state/pr-state/pr-24459.json
index 242a48d7d3b..a8fe46943bc 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-24459.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-24459.json
@@ -1,7 +1,10 @@
{
- "commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "lukecwik",
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[beam] branch master updated: [CdapIO] Add readme for CdapIO. Update readme for SparkReceiverIO. (#23959)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new a9dcf95453d [CdapIO] Add readme for CdapIO. Update readme for SparkReceiverIO. (#23959) a9dcf95453d is described below commit a9dcf95453d43621ef3ce63782f7cd6c47399bb6 Author: Vitaly Terentyev AuthorDate: Mon Dec 5 14:45:31 2022 +0400 [CdapIO] Add readme for CdapIO. Update readme for SparkReceiverIO. (#23959) * Add README for CdapIO. Update README for SparkReceiverIO. * Set export javadoc true for cdap and sparkreceiver * Updates to CdapIO and SparkReceiverIO readmes * Add manual how to add support for Batch and Streaming Cdap plugins. * Add manual how to add support for Spark Receiver. * Fix whitespace * updates to CDAP and SparkReceiver readme files * Updated CDAP readme * Fix links in readme Co-authored-by: Alex Kosolapov --- sdks/java/io/cdap/README.md | 145 ++ sdks/java/io/cdap/build.gradle| 1 - sdks/java/io/sparkreceiver/2/README.md| 40 +++-- sdks/java/io/sparkreceiver/2/build.gradle | 1 - 4 files changed, 180 insertions(+), 7 deletions(-) diff --git a/sdks/java/io/cdap/README.md b/sdks/java/io/cdap/README.md new file mode 100644 index 000..5269204c1d1 --- /dev/null +++ b/sdks/java/io/cdap/README.md @@ -0,0 +1,145 @@ + + +# CdapIO +CdapIO provides I/O transforms for [CDAP](https://cdap.io/) plugins. + +## What is CDAP? + +[CDAP](https://cdap.io/) is an application platform for building and managing data applications in hybrid and multi-cloud environments. +It enables developers, business analysts, and data scientists to use a visual rapid development environment and utilize common patterns, +data, and application abstractions to accelerate the development of data applications, addressing a broader range of real-time and batch use cases. + +[CDAP plugins](https://github.com/data-integrations) types: +- Batch source +- Batch sink +- Streaming source + +To learn more about CDAP plugins please see [io.cdap.cdap.api.annotation.Plugin](https://javadoc.io/static/io.cdap.cdap/cdap-api/6.7.2/io/cdap/cdap/api/annotation/Plugin.html) and [Data Integrations](https://github.com/data-integrations) plugins repository. + +## CDAP Batch plugins support in CDAP IO + +CdapIO supports CDAP Batch plugins based on Hadoop [InputFormat](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/InputFormat.html) and [OutputFormat](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapred/OutputFormat.html). +CDAP batch plugins support is implemented using [HadoopFormatIO](https://beam.apache.org/documentation/io/built-in/hadoop/). + +CdapIO currently supports the following CDAP Batch plugins by referencing `CDAP plugin` class: +* [Hubspot Batch Source](https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/source/batch/HubspotBatchSource.java) +* [Hubspot Batch Sink](https://github.com/data-integrations/hubspot/blob/develop/src/main/java/io/cdap/plugin/hubspot/sink/batch/HubspotBatchSink.java) +* [Salesforce Batch Source](https://github.com/data-integrations/salesforce/blob/develop/src/main/java/io/cdap/plugin/salesforce/plugin/source/batch/SalesforceBatchSource.java) +* [Salesforce Batch Sink](https://github.com/data-integrations/salesforce/blob/develop/src/main/java/io/cdap/plugin/salesforce/plugin/sink/batch/SalesforceBatchSink.java) +* [ServiceNow Batch Source](https://github.com/data-integrations/servicenow-plugins/blob/develop/src/main/java/io/cdap/plugin/servicenow/source/ServiceNowSource.java) +* [Zendesk Batch Source](https://github.com/data-integrations/zendesk/blob/develop/src/main/java/io/cdap/plugin/zendesk/source/batch/ZendeskBatchSource.java) + +It means that all these plugins can be used like this: +``CdapIO.withCdapPluginClass(HubspotBatchSource.class)`` + +### Requirements for Cdap Batch plugins + +CDAP Batch plugin should be based on `HadoopFormat` implementation. + +### How to add support for a new CDAP Batch plugin + +To add CdapIO support for a new CDAP Batch [Plugin](src/main/java/org/apache/beam/sdk/io/cdap/Plugin.java) perform the following steps: +1. Find CDAP plugin artifacts in the Maven Central repository. *Example:* [Hubspot plugin Maven repository](https://mvnrepository.com/artifact/io.cdap/hubspot-plugins/1.0.0). *Note:* To add a custom CDAP plugin, please follow [Sonatype publishing guidelines](https://central.sonatype.org/publish/). +2. Add the CDAP plugin Maven dependency to the `build.gradle` file. *Example:* ``implementation "io.cdap:hubspot-plugins:1.0.0"``. +3. Here are two ways of using CDAP batch plugin with CdapIO: + 1. Using `Plugin.createBatch()` method. Pass Cdap Plugin clas
