[beam] branch nightly-refs/heads/master updated (7bb7fab -> 8ed4763)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7bb7fab [release-23.6.0] Fix JIRA link for 2.36 blog (#16771) add 56e5c2c [BEAM-13647] Use role for Go worker binary. (#16729) add 13a8001 [BEAM-13840] Fix usage of legacy rawtypes in AWS modules add 8c0fa37 Merge pull request #16756: [BEAM-13840] Fix usage of legacy rawtypes in AWS modules add c691d12 [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1 add d1b8e56 Merge pull request #16755: [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1 add ae8c4fe [BEAM-13606] Fail bundles with failed BigTable mutations (#16751) add 60b36c2 BEAM-13439 Type annotation for ptransform_fn add c14d24d BEAM-13439 Type annotation for ptransform_fn #16517 add e3dbc5d Output successful rows from BQ Streaming Inserts add 6e050bc Merge pull request #16768 from Output successful rows from BQ Streaming Inserts add d192b62 [BEAM-13246] Add support for S3 Bucket Key at the object level (AWS Sdk v2). add 963c04a Merge pull request #16714: [BEAM-13246] Add support for S3 Bucket Key at the object level (AWS Sdk v2) add 52eeb65 Update README.md add cfeb1eb Merge branch 'apache:master' into master add 6999a51 Update README with latest PreCommit Jobs add b19e024 Update Postcommit jobs with latest jobs add 8425eb1 Update Performace job tests in readme add 9391007 update load job tests with latest updates add afad9aa update other jobs test with latest updates add c8d2ee0 mismatch links fix add 78fd7e6 update trigger phrase for some postCommit jobs add 16ac126 correct trigger phrases in readme add 017b80a Merge branch 'apache:master' into master add 639f6b4 Merge branch 'master' of github.com:MarcoRob/beam into BEAM-12650 add 1c7fd7a add pending jobs to readme add 4b833fa Update README.md add 7d39b33 fix broken links in jobs & remove the invalid ones add f210d1c Merge pull request #16610 from MarcoRob/BEAM-12650 add b09f51a [BEAM-12914] Add missing 3.9 opcodes to type inference. (#16761) add b14010d [BEAM-13321] Initial BigQueryIO externalization. (#16489) add f2c36cf [BEAM-13193] Enable process bundle response elements embedding in Java SDK Harness (#16769) add e1ef9c4 [BEAM-13830] added a debeziumio_expansion_addr flag to GoSDK (#16780) add fc00b96 Apply spotless. (#16783) add 0881676 Update Dataflow Python dev container images. add d37d030 Merge pull request #16782 from ibzib/py0208 add 87e7e28 [BEAM-13858] Fix broken github action on :sdks:go:examples:wordCount (#16785) add fd357fd [BEAM-13732] Go SDK BigQuery IO wrapper. Initial implementation. (#16598) add 6fe157a [BEAM-13732] Add example for Go BigQuery IO wrapper. (#16786) add 0a0dc72 Add java 17 to changes add 502b503 add jira for runner v2 add 8ed4763 Merge pull request #16779 from kileys/changes No new revisions were added by this update. Summary of changes: .test-infra/jenkins/README.md | 82 +-- CHANGES.md | 4 +- .../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- .../pipeline/src/main/proto/beam_runner_api.proto | 6 + sdks/go/container/boot.go | 16 +- sdks/go/examples/build.gradle | 1 + sdks/go/examples/kafka/taxi.go | 47 ++-- sdks/go/examples/xlang/bigquery/wordcount.go | 220 + sdks/go/pkg/beam/artifact/materialize.go | 5 +- sdks/go/pkg/beam/core/runtime/graphx/translate.go | 48 +++- .../pkg/beam/core/runtime/graphx/translate_test.go | 165 + sdks/go/pkg/beam/io/xlang/bigqueryio/bigquery.go | 267 + sdks/go/pkg/beam/io/xlang/schemaio/schemaio.go | 119 + .../go/pkg/beam/io/xlang/schemaio/schemaio_test.go | 80 ++ .../beam/runners/dataflow/dataflowlib/execute.go | 16 +- .../pkg/beam/runners/dataflow/dataflowlib/stage.go | 18 +- .../beam/runners/universal/runnerlib/execute.go| 64 +++-- .../pkg/beam/runners/universal/runnerlib/stage.go | 4 +- sdks/go/test/build.gradle | 3 + sdks/go/test/integration/flags.go | 4 + .../integration/io/xlang/debezium/debezium_test.go | 6 +- sdks/go/test/run_validatesrunner_tests.sh | 26 +- .../schemaio-expansion-service/build.gradle| 2 + .../sdk/fn/data/BeamFnDataOutboundAggregator.java | 40 ++- .../sdk/fn/data/BeamFnDataOutboundObserver.java| 6 +- .../fn/data/BeamFnDataOutboundAggregatorTest.java | 43 ++-- .../beam/fn/harness/PTransformRunnerFactory.java | 4 + .../fn/harness/control/ProcessBundleHandler.java | 94 ++-- .../beam/fn/harness/data/
[beam] branch master updated: [BEAM-13193] Allow BeamFnDataOutboundObserver to flush elements. (#16778)
This is an automated email from the ASF dual-hosted git repository.
yichi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3c6fb0d [BEAM-13193] Allow BeamFnDataOutboundObserver to flush
elements. (#16778)
3c6fb0d is described below
commit 3c6fb0d93132c89c730e444349949a1bd1eb1fad
Author: Yichi Zhang
AuthorDate: Tue Feb 8 19:58:29 2022 -0800
[BEAM-13193] Allow BeamFnDataOutboundObserver to flush elements. (#16778)
---
.../apache/beam/sdk/fn/data/BeamFnDataOutboundAggregator.java | 11 +--
.../apache/beam/sdk/fn/data/BeamFnDataOutboundObserver.java | 4 +++-
2 files changed, 8 insertions(+), 7 deletions(-)
diff --git
a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregator.java
b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregator.java
index b1b88c9..45c9daa 100644
---
a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregator.java
+++
b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundAggregator.java
@@ -107,8 +107,7 @@ public class BeamFnDataOutboundAggregator {
.setDaemon(true)
.setNameFormat("DataBufferOutboundFlusher-thread")
.build())
- .scheduleAtFixedRate(
- this::periodicFlush, timeLimit, timeLimit,
TimeUnit.MILLISECONDS);
+ .scheduleAtFixedRate(this::flush, timeLimit, timeLimit,
TimeUnit.MILLISECONDS);
}
}
@@ -160,7 +159,7 @@ public class BeamFnDataOutboundAggregator {
return receiver;
}
- private void flush() throws IOException {
+ private void flushInternal() {
if (bytesWrittenSinceFlush == 0) {
return;
}
@@ -265,10 +264,10 @@ public class BeamFnDataOutboundAggregator {
return bufferedElements;
}
- private void periodicFlush() {
+ void flush() {
try {
synchronized (flushLock) {
-flush();
+flushInternal();
}
} catch (Throwable t) {
throw new RuntimeException(t);
@@ -348,7 +347,7 @@ public class BeamFnDataOutboundAggregator {
perBundleByteCount += delta;
perBundleElementCount += 1;
if (bytesWrittenSinceFlush > sizeLimit) {
-flush();
+flushInternal();
}
}
diff --git
a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundObserver.java
b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundObserver.java
index 5baf88a..763f4bd 100644
---
a/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundObserver.java
+++
b/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataOutboundObserver.java
@@ -73,7 +73,9 @@ public class BeamFnDataOutboundObserver implements
CloseableFnDataReceiver
}
@Override
- public void flush() throws IOException {}
+ public void flush() throws IOException {
+aggregator.flush();
+ }
@Override
public void accept(T t) throws Exception {
[beam] 01/01: Update license_script.sh
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch tvalentyn-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 310c53aa8da45fa54bf32055a4ef37160590cc95
Author: tvalentyn
AuthorDate: Tue Feb 8 19:00:25 2022 -0800
Update license_script.sh
---
sdks/java/container/license_scripts/license_script.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/sdks/java/container/license_scripts/license_script.sh
b/sdks/java/container/license_scripts/license_script.sh
index c2a6540..979a74a 100755
--- a/sdks/java/container/license_scripts/license_script.sh
+++ b/sdks/java/container/license_scripts/license_script.sh
@@ -40,7 +40,7 @@ if [ -d "$DOWNLOAD_DIR" ]; then rm -rf "$DOWNLOAD_DIR" ; fi
mkdir -p "$DOWNLOAD_DIR"
cp -r "${EXISTING_LICENSE_DIR}"/*.jar "${DOWNLOAD_DIR}"
-$PYTHON -m venv ${ENV_DIR} && . ${ENV_DIR}/bin/activate
+$PYTHON -m venv --clear ${ENV_DIR} && . ${ENV_DIR}/bin/activate
pip install --retries 10 --upgrade pip setuptools wheel
# install packages
@@ -52,8 +52,8 @@ FLAGS="--license_index=${INDEX_FILE} \
--dep_url_yaml=${SCRIPT_DIR}/dep_urls_java.yaml \
--manual_license_path=${SCRIPT_DIR}/manual_licenses"
-echo "Executing ${ENV_DIR}/bin/python ${SCRIPT_DIR}/pull_licenses_java.py
$FLAGS"
-"${ENV_DIR}/bin/python" "${SCRIPT_DIR}/pull_licenses_java.py" $FLAGS
+echo "Executing python ${SCRIPT_DIR}/pull_licenses_java.py $FLAGS"
+python "${SCRIPT_DIR}/pull_licenses_java.py" $FLAGS
# If this script is running, it is assumed that outputs are out of date and
should be cleared and rewritten
if [ -d "$DEST_DIR" ]; then rm -rf "$DEST_DIR"; fi
[beam] branch tvalentyn-patch-1 created (now 310c53a)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git. at 310c53a Update license_script.sh This branch includes the following new commits: new 310c53a Update license_script.sh The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated (f36b6e1 -> 06de99b)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f36b6e1 Update CHANGES.md with Go SDK milestones. (#16787) add 5b9f6be [BEAM-13732] Switch x-lang BigQueryIO expansion service to GCP one. add 06de99b Merge pull request #16784: [BEAM-13732] Switch x-lang BigQueryIO expansion service. No new revisions were added by this update. Summary of changes: sdks/java/extensions/schemaio-expansion-service/build.gradle | 2 -- sdks/java/io/google-cloud-platform/expansion-service/build.gradle | 8 2 files changed, 8 insertions(+), 2 deletions(-)
[beam] branch master updated (8ed4763 -> f36b6e1)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8ed4763 Merge pull request #16779 from kileys/changes add f36b6e1 Update CHANGES.md with Go SDK milestones. (#16787) No new revisions were added by this update. Summary of changes: CHANGES.md | 8 1 file changed, 8 insertions(+)
[beam] branch lostluck-changemd37 updated (309bea9 -> 8bfa7c1)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-changemd37 in repository https://gitbox.apache.org/repos/asf/beam.git. from 309bea9 rm ws add 8bfa7c1 fix links No new revisions were added by this update. Summary of changes: CHANGES.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated (6fe157a -> 8ed4763)
This is an automated email from the ASF dual-hosted git repository. kileysok pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6fe157a [BEAM-13732] Add example for Go BigQuery IO wrapper. (#16786) new 0a0dc72 Add java 17 to changes new 502b503 add jira for runner v2 new 8ed4763 Merge pull request #16779 from kileys/changes The 34565 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
[beam] branch lostluck-changemd37 updated (9f61139 -> 309bea9)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-changemd37 in repository https://gitbox.apache.org/repos/asf/beam.git. from 9f61139 Update CHANGES.md add 309bea9 rm ws No new revisions were added by this update. Summary of changes: CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch lostluck-changemd37 created (now 9f61139)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch lostluck-changemd37 in repository https://gitbox.apache.org/repos/asf/beam.git. at 9f61139 Update CHANGES.md No new revisions were added by this update.
[beam] branch master updated: [BEAM-13732] Add example for Go BigQuery IO wrapper. (#16786)
This is an automated email from the ASF dual-hosted git repository.
danoliveira pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 6fe157a [BEAM-13732] Add example for Go BigQuery IO wrapper. (#16786)
6fe157a is described below
commit 6fe157aa3857dc44fc042bdb2d45a9fbf2dbaf1b
Author: Daniel Oliveira
AuthorDate: Tue Feb 8 17:42:59 2022 -0800
[BEAM-13732] Add example for Go BigQuery IO wrapper. (#16786)
Also does some polish and fixup to the Kafka taxi example to fix little
nits that were noticed while adding this new example.
---
sdks/go/examples/kafka/taxi.go | 47 +++---
sdks/go/examples/xlang/bigquery/wordcount.go | 220 +++
2 files changed, 245 insertions(+), 22 deletions(-)
diff --git a/sdks/go/examples/kafka/taxi.go b/sdks/go/examples/kafka/taxi.go
index 2c9c3c2..aeb539b 100644
--- a/sdks/go/examples/kafka/taxi.go
+++ b/sdks/go/examples/kafka/taxi.go
@@ -22,7 +22,9 @@
//
// Running this example requires a Kafka cluster accessible to the runner, and
// a cross-language expansion service that can expand Kafka read and write
-// transforms.
+// transforms. An address to a persistent expansion service can be provided as
+// a flag, or if none is specified then the SDK will attempt to automatically
+// start an appropriate expansion service.
//
// Setting Up a Kafka Cluster
//
@@ -34,22 +36,24 @@
//
// Running an Expansion Server
//
-// These instructions will cover running the Java IO Expansion Service, and
-// therefore requires a JDK installation in a version supported by Beam.
-// Depending on whether you are running this from a numbered Beam release, or a
-// development environment, there are two sources you may use for the Expansion
-// service.
+// If the automatic expansion service functionality is not available for your
+// environment, or if you want improved performance, you will need to start a
+// persistent expansion service. These instructions will cover running the Java
+// IO Expansion Service, and therefore requires a JDK installation in a version
+// supported by Beam. Depending on whether you are running this from a numbered
+// Beam release, or a development environment, there are two sources you may
+// use for the Expansion service.
//
// Numbered release: The expansion service jar is vendored as module
-// org.apache.beam:beam-sdks-java-io-expansion-service in Maven Repository.
-// This jar can be executed directly with the following command:
+// org.apache.beam:beam-sdks-java-io-expansion-service in Maven Repository.
+// This jar can be executed directly with the following command:
// `java -jar `
// Development env: This requires that the JAVA_HOME environment variable
-// points to your JDK installation. From the root `beam/` directory of the
-// Apache Beam repository, the jar can be built (or built and run) with the
-// following commands:
-// Build: ./gradlew :sdks:java:io:expansion-service:build
-// Build and Run: ./gradlew
:sdks:java:io:expansion-service:runExpansionService
-PconstructionService.port=
+// points to your JDK installation. From the root `beam/` directory of the
+// Apache Beam repository, the jar can be built (or built and run) with the
+// following commands:
+// Build: ./gradlew :sdks:java:io:expansion-service:build
+// Build and Run: ./gradlew
:sdks:java:io:expansion-service:runExpansionService
-PconstructionService.port=
//
// Running the Example on GCP
//
@@ -64,7 +68,7 @@
// export JOB_NAME="kafka-taxi-`date +%Y%m%d-%H%M%S`"
// export BOOTSTRAP_SERVERS="123.45.67.89:1234"
// export EXPANSION_ADDR="localhost:1234"
-// go run ./sdks/go/examples/kafka/types/types.go \
+// go run ./sdks/go/examples/kafka/taxi.go \
// --runner=DataflowRunner \
// --temp_location=$TEMP_LOCATION \
// --staging_location=$STAGING_LOCATION \
@@ -72,7 +76,6 @@
// --region=$REGION \
// --job_name="${JOB_NAME}" \
// --bootstrap_servers=$BOOTSTRAP_SERVER \
-// --experiments=use_portable_job_submission,use_runner_v2 \
// --expansion_addr=$EXPANSION_ADDR
//
// Running the Example From a Git Clone
@@ -91,9 +94,11 @@
// accessible locally.
//
// Additionally, you must provide the location of your custom container to the
-// pipeline with the --sdk_harness_container_image_override flag. For example:
+// pipeline with the --sdk_harness_container_image_override flag for Java, or
+// --environment_config flag for Go. For example:
//
-//
--sdk_harness_container_image_override=".*java.*,${DOCKER_ROOT}/beam_java8_sdk:latest"
+//
--sdk_harness_container_image_override=".*java.*,${DOCKER_ROOT}/beam_java8_sdk:latest"
\
+// --environment_config=${DOCKER_ROOT}/beam_go_sdk:latest
package main
import (
@@ -111,9 +116,10 @@ import (
)
var (
- expansionAddr= flag.String("expansion_addr", "", "Addre
[beam] branch master updated: [BEAM-13732] Go SDK BigQuery IO wrapper. Initial implementation. (#16598)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new fd357fd [BEAM-13732] Go SDK BigQuery IO wrapper. Initial implementation. (#16598) fd357fd is described below commit fd357fd1cb200a43db44a232823b3bd5b71c6494 Author: Daniel Oliveira AuthorDate: Tue Feb 8 17:34:33 2022 -0800 [BEAM-13732] Go SDK BigQuery IO wrapper. Initial implementation. (#16598) --- sdks/go/pkg/beam/io/xlang/bigqueryio/bigquery.go | 267 + sdks/go/pkg/beam/io/xlang/schemaio/schemaio.go | 119 + .../go/pkg/beam/io/xlang/schemaio/schemaio_test.go | 80 ++ 3 files changed, 466 insertions(+) diff --git a/sdks/go/pkg/beam/io/xlang/bigqueryio/bigquery.go b/sdks/go/pkg/beam/io/xlang/bigqueryio/bigquery.go new file mode 100644 index 000..19f2d62 --- /dev/null +++ b/sdks/go/pkg/beam/io/xlang/bigqueryio/bigquery.go @@ -0,0 +1,267 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +//http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package bigqueryio contains cross-language functionality for using Google Cloud BigQuery +// (https://cloud.google.com/bigquery). These transforms only work on runners that support +// cross-language transforms. +// +// Setup +// +// Transforms specified here are cross-language transforms implemented in a +// different SDK (listed below). During pipeline construction, the Go SDK will +// need to connect to an expansion service containing information on these +// transforms in their native SDK. If an expansion service address is not +// provided, an appropriate expansion service will be automatically started; +// however this is slower than having a persistent expansion service running. +// +// To use a persistent expansion service, it must be run as a separate process +// accessible during pipeline construction. The address of that process must be +// passed to the transforms in this package. +// +// The version of the expansion service should match the version of the Beam SDK +// being used. For numbered releases of Beam, these expansions services are +// released to the Maven repository as modules. For development versions of +// Beam, it is recommended to build and run it from source using Gradle. +// +// Current supported SDKs, including expansion service modules and reference +// documentation: +// * Java +// - Vendored Module: beam-sdks-java-extensions-schemaio-expansion-service +// - Run via Gradle: ./gradlew :sdks:java:extensions:schemaio-expansion-service:runExpansionService +// - Reference Class: org.apache.beam.sdk.io.gcp.bigquery.BigQuerySchemaIOProvider and +// org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO +// +// Type Conversions +// +// Elements are read from and written to BigQuery by first converting to a Beam schema Row type +// before converting to BigQuery compatible types. The following table lists all BigQuery types +// currently supported, and how they convert to Beam schema and Go types. +// ++--+-+ +// | BigQuery Standard SQL Type | Beam Schema Type | Go Type | +// ++--+-+ +// | BOOLEAN| BOOLEAN | bool| +// | INT64 | INT64| int64 | +// | FLOAT64| DOUBLE | float64 | +// | BYTES | BYTES| []byte | +// | STRING | STRING | string | +// | ARRAY | ARRAY| Special: slice | +// | STRUCT | ROW | Special: struct | +// ++--+-+ +// +// Array types are inferred from slice fields. For example, []int64 is equivalent to BigQuery's +// ARRAY. Struct types are inferred from nested structs in Go. +// +// Additionally, BigQuery schema fields can have a mode assigned to specify whether the field is +// Nullable, Required, or Repeated. In Go, Nullable fields are represent
[beam] branch master updated (d37d030 -> 87e7e28)
This is an automated email from the ASF dual-hosted git repository. yichi pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d37d030 Merge pull request #16782 from ibzib/py0208 add 87e7e28 [BEAM-13858] Fix broken github action on :sdks:go:examples:wordCount (#16785) No new revisions were added by this update. Summary of changes: sdks/go/examples/build.gradle | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated: Update Dataflow Python dev container images.
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 0881676 Update Dataflow Python dev container images. new d37d030 Merge pull request #16782 from ibzib/py0208 0881676 is described below commit 0881676b11ea52cfe69b699b45f9519d33f14995 Author: Kyle Weaver AuthorDate: Tue Feb 8 11:21:46 2022 -0800 Update Dataflow Python dev container images. --- sdks/python/apache_beam/runners/dataflow/internal/names.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py b/sdks/python/apache_beam/runners/dataflow/internal/names.py index be6a2c0..e99048b 100644 --- a/sdks/python/apache_beam/runners/dataflow/internal/names.py +++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py @@ -36,10 +36,10 @@ SERIALIZED_SOURCE_KEY = 'serialized_source' # Update this version to the next version whenever there is a change that will # require changes to legacy Dataflow worker execution environment. -BEAM_CONTAINER_VERSION = 'beam-master-20220126' +BEAM_CONTAINER_VERSION = 'beam-master-20220208' # Update this version to the next version whenever there is a change that # requires changes to SDK harness container or SDK harness launcher. -BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20220126' +BEAM_FNAPI_CONTAINER_VERSION = 'beam-master-20220208' DATAFLOW_CONTAINER_IMAGE_REPOSITORY = 'gcr.io/cloud-dataflow/v1beta3'
[beam] branch master updated (e1ef9c4 -> fc00b96)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e1ef9c4 [BEAM-13830] added a debeziumio_expansion_addr flag to GoSDK (#16780) add fc00b96 Apply spotless. (#16783) No new revisions were added by this update. Summary of changes: .../beam/sdk/io/gcp/bigquery/BigQuerySchemaIOProvider.java | 13 ++--- 1 file changed, 6 insertions(+), 7 deletions(-)
[beam] branch master updated (f2c36cf -> e1ef9c4)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f2c36cf [BEAM-13193] Enable process bundle response elements embedding in Java SDK Harness (#16769) add e1ef9c4 [BEAM-13830] added a debeziumio_expansion_addr flag to GoSDK (#16780) No new revisions were added by this update. Summary of changes: sdks/go/test/build.gradle | 3 +++ sdks/go/test/integration/flags.go | 4 .../integration/io/xlang/debezium/debezium_test.go | 6 ++--- sdks/go/test/run_validatesrunner_tests.sh | 26 +++--- sdks/java/io/expansion-service/build.gradle| 2 -- 5 files changed, 33 insertions(+), 8 deletions(-)
[beam] branch master updated (b14010d -> f2c36cf)
This is an automated email from the ASF dual-hosted git repository. yichi pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b14010d [BEAM-13321] Initial BigQueryIO externalization. (#16489) add f2c36cf [BEAM-13193] Enable process bundle response elements embedding in Java SDK Harness (#16769) No new revisions were added by this update. Summary of changes: .../sdk/fn/data/BeamFnDataOutboundAggregator.java | 40 +++-- .../sdk/fn/data/BeamFnDataOutboundObserver.java| 6 +- .../fn/data/BeamFnDataOutboundAggregatorTest.java | 43 ++ .../beam/fn/harness/PTransformRunnerFactory.java | 4 + .../fn/harness/control/ProcessBundleHandler.java | 94 + .../beam/fn/harness/data/BeamFnDataClient.java | 11 ++- .../beam/fn/harness/data/BeamFnDataGrpcClient.java | 7 +- .../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 3 +- .../beam/fn/harness/FnApiDoFnRunnerTest.java | 2 +- .../PTransformRunnerFactoryTestContext.java| 10 ++- .../harness/control/ProcessBundleHandlerTest.java | 95 -- .../fn/harness/data/BeamFnDataGrpcClientTest.java | 5 +- 12 files changed, 258 insertions(+), 62 deletions(-)
[beam] branch master updated (b09f51a -> b14010d)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b09f51a [BEAM-12914] Add missing 3.9 opcodes to type inference. (#16761) add b14010d [BEAM-13321] Initial BigQueryIO externalization. (#16489) No new revisions were added by this update. Summary of changes: .../schemaio-expansion-service/build.gradle| 2 + .../io/gcp/bigquery/BigQuerySchemaIOProvider.java | 216 + 2 files changed, 218 insertions(+) create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaIOProvider.java
[beam] branch master updated (f210d1c -> b09f51a)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f210d1c Merge pull request #16610 from MarcoRob/BEAM-12650 add b09f51a [BEAM-12914] Add missing 3.9 opcodes to type inference. (#16761) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/typehints/opcodes.py | 86 +- .../apache_beam/typehints/trivial_inference.py | 9 +++ .../typehints/trivial_inference_test.py| 43 +-- 3 files changed, 128 insertions(+), 10 deletions(-)
[beam] branch master updated (963c04a -> f210d1c)
This is an automated email from the ASF dual-hosted git repository. kileysok pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 963c04a Merge pull request #16714: [BEAM-13246] Add support for S3 Bucket Key at the object level (AWS Sdk v2) new 52eeb65 Update README.md new cfeb1eb Merge branch 'apache:master' into master new 6999a51 Update README with latest PreCommit Jobs new b19e024 Update Postcommit jobs with latest jobs new 8425eb1 Update Performace job tests in readme new 9391007 update load job tests with latest updates new afad9aa update other jobs test with latest updates new c8d2ee0 mismatch links fix new 78fd7e6 update trigger phrase for some postCommit jobs new 16ac126 correct trigger phrases in readme new 017b80a Merge branch 'apache:master' into master new 639f6b4 Merge branch 'master' of github.com:MarcoRob/beam into BEAM-12650 new 1c7fd7a add pending jobs to readme new 4b833fa Update README.md new 7d39b33 fix broken links in jobs & remove the invalid ones new f210d1c Merge pull request #16610 from MarcoRob/BEAM-12650 The 34552 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .test-infra/jenkins/README.md | 82 ++- 1 file changed, 65 insertions(+), 17 deletions(-)
[beam] branch master updated (6e050bc -> 963c04a)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6e050bc Merge pull request #16768 from Output successful rows from BQ Streaming Inserts add d192b62 [BEAM-13246] Add support for S3 Bucket Key at the object level (AWS Sdk v2). add 963c04a Merge pull request #16714: [BEAM-13246] Add support for S3 Bucket Key at the object level (AWS Sdk v2) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/sdk/io/aws2/options/S3Options.java | 7 +++ .../apache/beam/sdk/io/aws2/s3/S3FileSystemConfiguration.java| 9 + .../org/apache/beam/sdk/io/aws2/s3/S3WritableByteChannel.java| 1 + .../test/java/org/apache/beam/sdk/io/aws2/s3/S3TestUtils.java| 7 ++- .../apache/beam/sdk/io/aws2/s3/S3WritableByteChannelTest.java| 5 + 5 files changed, 28 insertions(+), 1 deletion(-)
[beam] branch master updated: Output successful rows from BQ Streaming Inserts
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e3dbc5d Output successful rows from BQ Streaming Inserts
new 6e050bc Merge pull request #16768 from Output successful rows from BQ
Streaming Inserts
e3dbc5d is described below
commit e3dbc5d90cc94cb4e04573d0634927b90a07e7f6
Author: Pablo Estrada
AuthorDate: Mon Feb 7 14:16:34 2022 -0800
Output successful rows from BQ Streaming Inserts
---
.../sdk/io/gcp/bigquery/BatchedStreamingWrite.java | 3 ++
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 39 ++
2 files changed, 28 insertions(+), 14 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchedStreamingWrite.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchedStreamingWrite.java
index f5ee5ea..015a5e0 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchedStreamingWrite.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchedStreamingWrite.java
@@ -274,6 +274,9 @@ class BatchedStreamingWrite
for (ValueInSingleWindow row : failedInserts) {
context.output(failedOutputTag, row.getValue(), row.getTimestamp(),
row.getWindow());
}
+ for (ValueInSingleWindow row : successfulInserts) {
+context.output(SUCCESSFUL_ROWS_TAG, row.getValue(),
row.getTimestamp(), row.getWindow());
+ }
reportStreamingApiLogging(options);
}
diff --git
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
index df34790..7b2df25 100644
---
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
+++
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java
@@ -1305,20 +1305,31 @@ public class BigQueryIOWriteTest implements
Serializable {
if (useStorageApi || !useStreaming) {
return;
}
-p.apply(
-Create.of(
-new SchemaPojo("a", 1),
-new SchemaPojo("b", 2),
-new SchemaPojo("c", 3),
-new SchemaPojo("d", 4)))
-.apply(
-BigQueryIO.write()
-.to("project-id:dataset-id.table-id")
-
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
-.withMethod(Method.STREAMING_INSERTS)
-.useBeamSchema()
-.withTestServices(fakeBqServices)
-.withoutValidation());
+WriteResult result =
+p.apply(
+Create.of(
+new SchemaPojo("a", 1),
+new SchemaPojo("b", 2),
+new SchemaPojo("c", 3),
+new SchemaPojo("d", 4)))
+.apply(
+BigQueryIO.write()
+.to("project-id:dataset-id.table-id")
+.withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
+.withMethod(Method.STREAMING_INSERTS)
+.useBeamSchema()
+.withTestServices(fakeBqServices)
+.withoutValidation());
+
+PAssert.that(result.getSuccessfulInserts())
+.satisfies(
+new SerializableFunction, Void>() {
+ @Override
+ public Void apply(Iterable input) {
+assertThat(Lists.newArrayList(input).size(), is(4));
+return null;
+ }
+});
p.run();
assertThat(
[beam] 01/01: BEAM-13439 Type annotation for ptransform_fn #16517
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit c14d24d0de17c559878707c21a1406f45027b721 Merge: ae8c4fe 60b36c2 Author: tvalentyn AuthorDate: Tue Feb 8 08:27:51 2022 -0800 BEAM-13439 Type annotation for ptransform_fn #16517 sdks/python/apache_beam/transforms/ptransform.py | 2 ++ 1 file changed, 2 insertions(+)
[beam] branch master updated (ae8c4fe -> c14d24d)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ae8c4fe [BEAM-13606] Fail bundles with failed BigTable mutations (#16751) add 60b36c2 BEAM-13439 Type annotation for ptransform_fn new c14d24d BEAM-13439 Type annotation for ptransform_fn #16517 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/transforms/ptransform.py | 2 ++ 1 file changed, 2 insertions(+)
[beam] branch master updated (d1b8e56 -> ae8c4fe)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d1b8e56 Merge pull request #16755: [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1 add ae8c4fe [BEAM-13606] Fail bundles with failed BigTable mutations (#16751) No new revisions were added by this update. Summary of changes: .../examples/cookbook/bigtableio_it_test.py| 8 -- sdks/python/apache_beam/io/gcp/bigtableio.py | 29 +- sdks/python/apache_beam/io/gcp/bigtableio_test.py | 7 +- 3 files changed, 35 insertions(+), 9 deletions(-)
[beam] branch master updated (8c0fa37 -> d1b8e56)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8c0fa37 Merge pull request #16756: [BEAM-13840] Fix usage of legacy rawtypes in AWS modules add c691d12 [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1 new d1b8e56 Merge pull request #16755: [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #16755: [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit d1b8e569fd651975f08823a3db49dbee56d491b5 Merge: 8c0fa37 c691d12 Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com> AuthorDate: Tue Feb 8 11:49:27 2022 +0100 Merge pull request #16755: [BEAM-13839] Upgrade zstd-jni to version 1.5.2-1 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (56e5c2c -> 8c0fa37)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 56e5c2c [BEAM-13647] Use role for Go worker binary. (#16729) add 13a8001 [BEAM-13840] Fix usage of legacy rawtypes in AWS modules add 8c0fa37 Merge pull request #16756: [BEAM-13840] Fix usage of legacy rawtypes in AWS modules No new revisions were added by this update. Summary of changes: .../beam/sdk/io/aws/dynamodb/DynamoDBIO.java | 29 +++--- .../java/org/apache/beam/sdk/io/aws/sns/SnsIO.java | 7 ++ .../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 29 +++--- .../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 5 ++-- 4 files changed, 34 insertions(+), 36 deletions(-)
