[beam] branch BEAM-9371-sideinput-java updated (6cec849 -> 0cc55d5)
This is an automated email from the ASF dual-hosted git repository. mwalenia pushed a change to branch BEAM-9371-sideinput-java in repository https://gitbox.apache.org/repos/asf/beam.git. from 6cec849 Fix spotbugs add 0cc55d5 Fix code style error No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/sdk/loadtests/SideInputLoadTest.java| 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
[beam] branch master updated (083edb4 -> d7df9ed)
This is an automated email from the ASF dual-hosted git repository. amaliujia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 083edb4 [BEAM-7926] Data-centric Interactive Part3 (#10731) add d7df9ed [BEAM-9240]: Check for Nullability in typesEqual() method of FieldType class (#10744) No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/schemas/Schema.java | 3 +++ .../org/apache/beam/sdk/schemas/SchemaTest.java| 25 +++--- .../sdk/extensions/sql/impl/rel/BeamCalcRel.java | 9 +--- .../extensions/sql/impl/utils/CalciteUtils.java| 5 + 4 files changed, 36 insertions(+), 6 deletions(-)
[beam] branch master updated (9fa98ce -> 083edb4)
This is an automated email from the ASF dual-hosted git repository. altay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9fa98ce [BEAM-9228] Support further partition for FnApi ListBuffer (#10847) add 083edb4 [BEAM-7926] Data-centric Interactive Part3 (#10731) No new revisions were added by this update. Summary of changes: .../runners/interactive/background_caching_job.py | 3 +- .../interactive/display/pcoll_visualization.py | 7 +- .../display/pcoll_visualization_test.py| 6 +- .../examples/Interactive Beam Example.ipynb| 57 +-- .../runners/interactive/interactive_beam.py| 104 - .../runners/interactive/interactive_beam_test.py | 30 ++ .../runners/interactive/interactive_runner.py | 11 ++- .../runners/interactive/pipeline_fragment.py | 13 ++- .../runners/interactive/pipeline_instrument.py | 7 +- .../interactive/pipeline_instrument_test.py| 32 +++ 10 files changed, 229 insertions(+), 41 deletions(-)
[beam] branch master updated (3169579 -> 9fa98ce)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3169579 Merge pull request #10935 from boyuanzz/type_hint add 9fa98ce [BEAM-9228] Support further partition for FnApi ListBuffer (#10847) No new revisions were added by this update. Summary of changes: .../runners/portability/fn_api_runner.py | 174 +++-- .../runners/portability/fn_api_runner_test.py | 17 -- 2 files changed, 125 insertions(+), 66 deletions(-)
[beam] branch master updated (7139216 -> 3169579)
This is an automated email from the ASF dual-hosted git repository. boyuanz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7139216 Merge pull request #10933 from boyuanzz/follow_up new c2dfa8d Clean up and add type-hints to SDF API new 43e0f64 Address comments new 1117f47 Address comments again new 3169579 Merge pull request #10935 from boyuanzz/type_hint The 25712 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/io/iobase.py | 4 sdks/python/apache_beam/runners/common.py| 29 ++-- sdks/python/apache_beam/runners/sdf_utils.py | 17 sdks/python/apache_beam/transforms/core.py | 1 + 4 files changed, 27 insertions(+), 24 deletions(-)
[beam] branch master updated: Update docstring of ManualWatermarkEstimator.set_watermark()
This is an automated email from the ASF dual-hosted git repository.
boyuanz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f2f0282 Update docstring of ManualWatermarkEstimator.set_watermark()
new 7139216 Merge pull request #10933 from boyuanzz/follow_up
f2f0282 is described below
commit f2f0282719194f31557d628d0fb1fc6a44d72c53
Author: Boyuan Zhang
AuthorDate: Fri Feb 21 15:17:41 2020 -0800
Update docstring of ManualWatermarkEstimator.set_watermark()
---
sdks/python/apache_beam/io/watermark_estimators.py | 17 +
1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/sdks/python/apache_beam/io/watermark_estimators.py
b/sdks/python/apache_beam/io/watermark_estimators.py
index 5a00ae7..65c4611 100644
--- a/sdks/python/apache_beam/io/watermark_estimators.py
+++ b/sdks/python/apache_beam/io/watermark_estimators.py
@@ -120,10 +120,19 @@ class ManualWatermarkEstimator(WatermarkEstimator):
return self._watermark
def set_watermark(self, timestamp):
-# Please call set_watermark after calling restriction_tracker.try_claim()
to
-# prevent advancing watermark early.
-# TODO(BEAM-7473): It's possible that getting a slightly stale watermark
-# when performing split.
+# pylint: disable=line-too-long
+
+"""Sets a timestamp before or at the timestamps of all future elements
+produced by the associated DoFn.
+
+This can be approximate. If records are output that violate this guarantee,
+they will be considered late, which will affect how they will be processed.
+See
https://beam.apache.org/documentation/programming-guide/#watermarks-and-late-data
+for more information on late data and how to handle it.
+
+However, this value should be as late as possible. Downstream windows may
+not be able to close until this watermark passes their end.
+"""
if not isinstance(timestamp, Timestamp):
raise ValueError('set_watermark expects a Timestamp as input')
if self._watermark and self._watermark > timestamp:
[beam] branch master updated: Support null fields in rows with ByteBuddy generated code.
This is an automated email from the ASF dual-hosted git repository.
reuvenlax pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b63bd59 Support null fields in rows with ByteBuddy generated code.
new 16fa12c Merge pull request #10926 from
reuvenlax/fix_bytebuddy_nullable
b63bd59 is described below
commit b63bd5928864f37d88746f48566e6ed56b8b6660
Author: Reuven Lax
AuthorDate: Thu Feb 20 21:46:34 2020 -0800
Support null fields in rows with ByteBuddy generated code.
---
.../beam/sdk/schemas/utils/AutoValueUtils.java | 3 +
.../beam/sdk/schemas/utils/AvroByteBuddyUtils.java | 3 +
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 290 +
.../beam/sdk/schemas/utils/ConvertHelpers.java | 3 +
.../beam/sdk/schemas/utils/JavaBeanUtils.java | 6 +
.../apache/beam/sdk/schemas/utils/POJOUtils.java | 9 +
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 44
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 54
.../beam/sdk/schemas/utils/TestJavaBeans.java | 199 ++
.../apache/beam/sdk/schemas/utils/TestPOJOs.java | 105
.../extensions/protobuf/ProtoByteBuddyUtils.java | 3 +
11 files changed, 612 insertions(+), 107 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AutoValueUtils.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AutoValueUtils.java
index 775fe65..0435cb8 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AutoValueUtils.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AutoValueUtils.java
@@ -44,6 +44,7 @@ import
org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.TypeConversionsFactory;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.asm.AsmVisitorWrapper;
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.method.MethodDescription.ForLoadedMethod;
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
@@ -62,6 +63,7 @@ import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.byte
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodInvocation;
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodReturn;
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.member.MethodVariableAccess;
+import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.ClassWriter;
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.matcher.ElementMatchers;
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
@@ -207,6 +209,7 @@ public class AutoValueUtils {
.intercept(
new BuilderCreateInstruction(types, setterMethods,
builderClass, buildMethod));
return builder
+ .visit(new
AsmVisitorWrapper.ForDeclaredMethods().writerFlags(ClassWriter.COMPUTE_FRAMES))
.make()
.load(ReflectHelpers.findClassLoader(),
ClassLoadingStrategy.Default.INJECTION)
.getLoaded()
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java
index 3bb7e8d..8957cf0 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroByteBuddyUtils.java
@@ -34,6 +34,7 @@ import
org.apache.beam.sdk.schemas.utils.ReflectUtils.ClassWithSchema;
import org.apache.beam.sdk.util.common.ReflectHelpers;
import org.apache.beam.sdk.values.TypeDescriptor;
import org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.ByteBuddy;
+import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.asm.AsmVisitorWrapper;
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.description.type.TypeDescription.ForLoadedType;
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.DynamicType;
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
@@ -43,6 +44,7 @@ import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.byte
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.collection.ArrayAccess;
import
org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.implementation.bytecode.constant.IntegerConstant;
import
org.apache.beam.vendor.bytebuddy.v
[beam] branch master updated: Bump google cloud bigquery to 1.24.0
This is an automated email from the ASF dual-hosted git repository. altay pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 915a98a Bump google cloud bigquery to 1.24.0 new 71134d2 Merge pull request #10929 from drubinstein/bump-google-cloud-bigquery 915a98a is described below commit 915a98a946ef4d483b3a7964aa8adabc910be3ce Author: David Rubinstein AuthorDate: Fri Feb 21 13:26:07 2020 -0500 Bump google cloud bigquery to 1.24.0 --- sdks/python/container/base_image_requirements.txt | 2 +- sdks/python/setup.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/sdks/python/container/base_image_requirements.txt b/sdks/python/container/base_image_requirements.txt index 47c5dd5..9499fe1 100644 --- a/sdks/python/container/base_image_requirements.txt +++ b/sdks/python/container/base_image_requirements.txt @@ -50,7 +50,7 @@ nose==1.3.7 google-apitools==0.5.28 googledatastore==7.0.2;python_version<"3.0" google-cloud-pubsub==0.39.1 -google-cloud-bigquery==1.17.0 +google-cloud-bigquery==1.24.0 proto-google-cloud-datastore-v1==0.90.4 google-cloud-bigtable==0.32.1 google-cloud-core==1.0.2 diff --git a/sdks/python/setup.py b/sdks/python/setup.py index dc4ca33..1cfcb2e 100644 --- a/sdks/python/setup.py +++ b/sdks/python/setup.py @@ -204,7 +204,7 @@ GCP_REQUIREMENTS = [ 'google-cloud-datastore>=1.7.1,<1.8.0', 'google-cloud-pubsub>=0.39.0,<1.1.0', # GCP packages required by tests -'google-cloud-bigquery>=1.6.0,<1.18.0', +'google-cloud-bigquery>=1.6.0,<=1.24.0', 'google-cloud-core>=0.28.1,<2', 'google-cloud-bigtable>=0.31.1,<1.1.0', 'google-cloud-dlp >=0.12.0,<=0.13.0',
[beam] branch master updated (aebc5a7 -> e9cbb8c)
This is an automated email from the ASF dual-hosted git repository. goenka pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from aebc5a7 [GoSDK] Fix race condition in statemgr & test (#10941) add e9cbb8c [BEAM-9347] Don't overwrite default runner harness for unified worker (#10919) No new revisions were added by this update. Summary of changes: .../runners/dataflow/internal/apiclient.py | 13 +-- .../runners/dataflow/internal/apiclient_test.py| 45 ++ 2 files changed, 55 insertions(+), 3 deletions(-)
[beam] branch master updated (b83d59e -> aebc5a7)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b83d59e Merge pull request #10936 from Hannah-Jiang/docker_images_migration add aebc5a7 [GoSDK] Fix race condition in statemgr & test (#10941) No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/core/runtime/harness/statemgr.go | 2 -- .../pkg/beam/core/runtime/harness/statemgr_test.go | 37 +- 2 files changed, 29 insertions(+), 10 deletions(-)
[beam] branch master updated: fix postcommit failure
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 88914cf fix postcommit failure new b83d59e Merge pull request #10936 from Hannah-Jiang/docker_images_migration 88914cf is described below commit 88914cf7c79ca185e2f67a03a7d1dc57372c6873 Author: Hannah Jiang AuthorDate: Fri Feb 21 16:48:19 2020 -0800 fix postcommit failure --- sdks/go/test/run_integration_tests.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/go/test/run_integration_tests.sh b/sdks/go/test/run_integration_tests.sh index e021ab9..78aef1a 100755 --- a/sdks/go/test/run_integration_tests.sh +++ b/sdks/go/test/run_integration_tests.sh @@ -125,7 +125,7 @@ fi # Build the container TAG=$(date +%Y%m%d-%H%M%S) -CONTAINER=us.gcr.io/$PROJECT/$USER/go_sdk +CONTAINER=us.gcr.io/$PROJECT/$USER/beam_go_sdk echo "Using container $CONTAINER" ./gradlew :sdks:go:container:docker -Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
[beam] branch master updated (903261c -> dfef2cd)
This is an automated email from the ASF dual-hosted git repository. udim pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 903261c Merge pull request #10939 from iemejia/BEAM-9342-bytebuddy-update add dfef2cd [BEAM-8280] Fix IOTypeHints origin traceback on partials (#10927) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/typehints/decorators.py | 3 ++- sdks/python/apache_beam/typehints/decorators_test_py3.py | 10 ++ 2 files changed, 12 insertions(+), 1 deletion(-)
[beam] branch master updated: [BEAM-9342] Upgrade vendored bytebuddy to version 1.10.8
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 17bb5e2 [BEAM-9342] Upgrade vendored bytebuddy to version 1.10.8
new 903261c Merge pull request #10939 from
iemejia/BEAM-9342-bytebuddy-update
17bb5e2 is described below
commit 17bb5e29a00362cb9fcc14bd93ad32eafacd9bcb
Author: Ismaël Mejía
AuthorDate: Thu Feb 20 12:58:52 2020 +0100
[BEAM-9342] Upgrade vendored bytebuddy to version 1.10.8
---
settings.gradle | 2 +-
vendor/{bytebuddy-1_9_3 => bytebuddy-1_10_8}/build.gradle | 8
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/settings.gradle b/settings.gradle
index a0f04f1..e8da75e 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -150,7 +150,7 @@ include ":sdks:python:test-suites:tox:py35"
include ":sdks:python:test-suites:tox:py36"
include ":sdks:python:test-suites:tox:py37"
include ":vendor:grpc-1_26_0"
-include ":vendor:bytebuddy-1_9_3"
+include ":vendor:bytebuddy-1_10_8"
include ":vendor:calcite-1_20_0"
include ":vendor:guava-26_0-jre"
include ":vendor:sdks-java-extensions-protobuf"
diff --git a/vendor/bytebuddy-1_9_3/build.gradle
b/vendor/bytebuddy-1_10_8/build.gradle
similarity index 84%
rename from vendor/bytebuddy-1_9_3/build.gradle
rename to vendor/bytebuddy-1_10_8/build.gradle
index 73bf327..f371380 100644
--- a/vendor/bytebuddy-1_9_3/build.gradle
+++ b/vendor/bytebuddy-1_10_8/build.gradle
@@ -18,17 +18,17 @@
plugins { id 'org.apache.beam.vendor-java' }
-description = "Apache Beam :: Vendored Dependencies :: ByteBuddy :: 1.9.3"
+description = "Apache Beam :: Vendored Dependencies :: ByteBuddy :: 1.10.8"
group = "org.apache.beam"
version = "0.1"
vendorJava(
- dependencies: ["net.bytebuddy:byte-buddy:1.9.3"],
+ dependencies: ["net.bytebuddy:byte-buddy:1.10.8"],
relocations: [
-"net.bytebuddy": "org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy"
+"net.bytebuddy": "org.apache.beam.vendor.bytebuddy.v1_10_8.net.bytebuddy"
],
groupId: group,
- artifactId: "beam-vendor-bytebuddy-1_9_3",
+ artifactId: "beam-vendor-bytebuddy-1_10_8",
version: version
)
[beam] branch master updated (58656c6 -> 2e0f431)
This is an automated email from the ASF dual-hosted git repository. reuvenlax pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 58656c6 Merge pull request #10909 [BEAM-9340] Populate requirements for Python DoFn properties. add 2e0f431 Merge pull request #10944: [BEAM-7274] optimize oneOf handling No new revisions were added by this update. Summary of changes: .../beam/sdk/schemas/utils/ByteBuddyUtils.java | 2 +- .../extensions/protobuf/ProtoByteBuddyUtils.java | 612 + 2 files changed, 521 insertions(+), 93 deletions(-)
[beam] branch master updated (c14a185 -> 58656c6)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c14a185 Merge pull request #10937 [BEAM-8019] Branch on having multiple environments. add 742c570 Add some standard requirement URNs to the protos. add 25dccdb [BEAM-9340] Populate requirements for Python DoFn properties. add 58656c6 Merge pull request #10909 [BEAM-9340] Populate requirements for Python DoFn properties. No new revisions were added by this update. Summary of changes: .../src/main/proto/beam_expansion_api.proto| 4 ++ .../pipeline/src/main/proto/beam_runner_api.proto | 48 +++-- sdks/python/apache_beam/pipeline.py| 7 ++- sdks/python/apache_beam/pipeline_test.py | 11 + sdks/python/apache_beam/portability/common_urns.py | 2 + sdks/python/apache_beam/runners/common.py | 13 ++ .../python/apache_beam/runners/pipeline_context.py | 11 - .../runners/portability/expansion_service.py | 3 +- .../runners/portability/fn_api_runner.py | 50 ++ sdks/python/apache_beam/transforms/core.py | 15 +-- sdks/python/apache_beam/transforms/external.py | 4 ++ 11 files changed, 157 insertions(+), 11 deletions(-)
[beam] branch master updated (10ade7b -> c14a185)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 10ade7b Merge pull request #10948: Add Dataflow Java11 ValidatesRunner badge to the PR template add 6a125bb [BEAM-8019] Branch on having multiple environments. add c14a185 Merge pull request #10937 [BEAM-8019] Branch on having multiple environments. No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-)
[beam] branch BEAM-9371-sideinput-java updated (befbd23 -> 6cec849)
This is an automated email from the ASF dual-hosted git repository. mwalenia pushed a change to branch BEAM-9371-sideinput-java in repository https://gitbox.apache.org/repos/asf/beam.git. from befbd23 [BEAM-9731] Add SideInputLoadTest implementation add 6cec849 Fix spotbugs No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/sdk/loadtests/SideInputLoadTest.java | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-)
[beam] 01/01: [BEAM-9731] Add SideInputLoadTest implementation
This is an automated email from the ASF dual-hosted git repository.
mwalenia pushed a commit to branch BEAM-9371-sideinput-java
in repository https://gitbox.apache.org/repos/asf/beam.git
commit befbd2361fbdf26e495eeb440f9fffdd93cb928c
Author: Michal Walenia
AuthorDate: Mon Feb 24 16:01:36 2020 +0100
[BEAM-9731] Add SideInputLoadTest implementation
---
.../beam/sdk/loadtests/SideInputLoadTest.java | 240 +
1 file changed, 240 insertions(+)
diff --git
a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/SideInputLoadTest.java
b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/SideInputLoadTest.java
new file mode 100644
index 000..a0b0365
--- /dev/null
+++
b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/SideInputLoadTest.java
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.loadtests;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import org.apache.beam.sdk.io.synthetic.SyntheticStep;
+import org.apache.beam.sdk.options.Default;
+import org.apache.beam.sdk.options.Description;
+import org.apache.beam.sdk.options.Validation;
+import org.apache.beam.sdk.testutils.metrics.ByteMonitor;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * Load test for operations involving side inputs.
+ *
+ * The purpose of this test is to measure cost of materialization or lookup
of side inputs. It
+ * uses synthetic sources and {@link SyntheticStep} which can be parametrized
to generate records
+ * with various sizes of keys and values, impose delays in the pipeline and
simulate other
+ * performance challenges.
+ *
+ * To run the test manually, use the following command:
+ *
+ *
+ * ./gradlew :sdks:java:testing:load-tests:run -PloadTest.args='
+ *--sourceOptions={"numRecords":2000, ...}
+ *--sideInputType=ITERABLE
+ *--accessPercentage=1
+ *--windowCount=200
+ *
+ */
+public class SideInputLoadTest extends LoadTest {
+
+ private static final String METRICS_NAMESPACE = "sideinput";
+ private static final Instant TIME = new Instant();
+
+ public SideInputLoadTest(String[] args) throws IOException {
+super(args, Options.class, METRICS_NAMESPACE);
+ }
+
+ @Override
+ void loadTest() throws IOException {
+Optional syntheticStep =
createStep(options.getStepOptions());
+PCollection> input =
+pipeline
+.apply(readFromSource(sourceOptions))
+.apply(ParDo.of(new AddTimestamps()))
+.apply("Collect start time metrics", ParDo.of(runtimeMonitor))
+.apply(ParDo.of(new ByteMonitor(METRICS_NAMESPACE,
"totalBytes.count")));
+
+performTestWithSideInput(
+input,
SideInputMaterializationType.valueOf(options.getSideInputType()),
syntheticStep);
+ }
+
+ private void performTestWithSideInput(
+ PCollection> input,
+ SideInputMaterializationType sideInputType,
+ Optional syntheticStep) {
+switch (sideInputType) {
+ case ITERABLE:
+performTestWithIterable(input, syntheticStep);
+break;
+ case HASHMAP:
+performTestWithHashMap(input, syntheticStep);
+break;
+ case LIST:
+performTestWithList(input, syntheticStep);
+break;
+}
+ }
+
+ private void performTestWithList(
+ PCollection> input, Optional
syntheticStep) {
+applyStepIfPresent(input, "Synthetic step", syntheticStep);
+PCollectionView>> sideInput =
input.apply(View.asList());
+input
+.apply(ParDo.of(new
SideInputTestWithList(sideInput)).withSideInputs(sideInput))
+.apply("Collect end time metrics", ParDo.of(runtimeMonitor));
+
[beam] branch BEAM-9371-sideinput-java created (now befbd23)
This is an automated email from the ASF dual-hosted git repository. mwalenia pushed a change to branch BEAM-9371-sideinput-java in repository https://gitbox.apache.org/repos/asf/beam.git. at befbd23 [BEAM-9731] Add SideInputLoadTest implementation This branch includes the following new commits: new befbd23 [BEAM-9731] Add SideInputLoadTest implementation The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated (a537ae6 -> 10ade7b)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a537ae6 [BEAM-9344] Add support for bundle finalization execution to the Beam Java SDK. (#10910) add 2d05c26 Add Dataflow Java11 ValidatesRunner badge to the PR template add 10ade7b Merge pull request #10948: Add Dataflow Java11 ValidatesRunner badge to the PR template No new revisions were added by this update. Summary of changes: .github/PULL_REQUEST_TEMPLATE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
