[beam] branch master updated: [BEAM-6225] Setup Jenkins Job to Run VR with ExecutableStage (#7271)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b7035c1 [BEAM-6225] Setup Jenkins Job to Run VR with ExecutableStage
(#7271)
b7035c1 is described below
commit b7035c1a098c526356c6fb33480b989ed037da0a
Author: Boyuan Zhang <36090911+boyua...@users.noreply.github.com>
AuthorDate: Fri Dec 14 13:47:48 2018 -0800
[BEAM-6225] Setup Jenkins Job to Run VR with ExecutableStage (#7271)
---
...unner_DataflowPortabilityExecutableStage.groovy | 54 ++
runners/google-cloud-dataflow-java/build.gradle| 48 +--
...aflowPortabilityExecutableStageUnsupported.java | 25 ++
.../apache/beam/sdk/testing/UsesSideInputs.java| 24 ++
.../org/apache/beam/sdk/testing/PAssertTest.java | 12 ++---
.../apache/beam/sdk/transforms/CombineTest.java| 30 ++--
.../apache/beam/sdk/transforms/FlattenTest.java| 3 +-
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 11 +++--
.../org/apache/beam/sdk/transforms/ParDoTest.java | 17 +++
.../apache/beam/sdk/transforms/ReshuffleTest.java | 11 +++--
.../beam/sdk/transforms/SplittableDoFnTest.java| 10 +++-
.../beam/sdk/transforms/join/CoGroupByKeyTest.java | 8 ++--
.../beam/sdk/transforms/windowing/WindowTest.java | 5 +-
.../sdk/transforms/windowing/WindowingTest.java| 9 ++--
14 files changed, 212 insertions(+), 55 deletions(-)
diff --git
a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy
b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy
new file mode 100644
index 000..62e7361
--- /dev/null
+++
b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+
+// This job runs the suite of ValidatesRunner tests against the Dataflow
+// runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage',
+ 'Run Dataflow Portability ExecutableStage ValidatesRunner', 'Google Cloud
Dataflow Runner PortabilityApi ExecutableStage ValidatesRunner Tests', this) {
+
+ description('Runs the ValidatesRunner suite on the Dataflow PortabilityApi
runner with ExecutableStage code path enabled.')
+
+ // Set common parameters. Sets a 3 hour timeout.
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 400)
+
+ // Publish all test results to Jenkins
+ publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+ }
+
+ // Gradle goals for this job.
+ steps {
+gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+
tasks(':beam-runners-google-cloud-dataflow-java:validatesRunnerFnApiWorkerExecutableStageTest')
+ // Increase parallel worker threads above processor limit since most
time is
+ // spent waiting on Dataflow jobs. ValidatesRunner tests on Dataflow are
slow
+ // because each one launches a Dataflow job with about 3 mins of
overhead.
+ // 3 x num_cores strikes a good balance between maxing out parallelism
without
+ // overloading the machines.
+ commonJobProperties.setGradleSwitches(delegate, 3 *
Runtime.runtime.availableProcessors())
+}
+ }
+
+ // [BEAM-6236] "use_executable_stage_bundle_execution" hasn't been rolled
out.
+ disabled()
+}
diff --git a/runners/google-cloud-dataflow-java/build.gradle
b/runners/google-cloud-dataflow-java/build.gradle
index c0b831c..9c6aaf4 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -241,17 +241,55 @@ task validatesRunnerFnApiWorkerTest(type: Test) {
}
}
+task validatesRunnerFnApiWorkerExecutableStageTest(type: Test) {
+group = "Verification"
+description "Validates Dataflow PortabilityApi runner"
+dependsOn
":beam-runners-google-cloud-dataflow-java-fn-api-worker:shadowJar"
+depends
[beam] branch master updated (5ec695b -> ec3f792)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 5ec695b [BEAM-6150] Superinterface for SerializableFunction allowing
declared exceptions (#7160)
add 4f90294 Remove Gradle from Jenkins job names
new ec3f792 Merge pull request #7286 from swegner/jenkins_gradle
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md| 4 ++--
...ob_PostCommit_Go_GradleBuild.groovy => job_PostCommit_Go.groovy} | 3 ++-
...ostCommit_Java_GradleBuild.groovy => job_PostCommit_Java.groovy} | 3 ++-
...GradleBuild.groovy => job_PostCommit_Java_PortabilityApi.groovy} | 3 ++-
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Apex.groovy | 5 ++---
.../jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy | 5 ++---
.../jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy| 3 ++-
.../jenkins/job_PostCommit_Java_ValidatesRunner_Gearpump.groovy | 5 ++---
...b_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy | 3 ++-
.../jenkins/job_PostCommit_Java_ValidatesRunner_Samza.groovy| 3 ++-
.../jenkins/job_PostCommit_Java_ValidatesRunner_Spark.groovy| 5 ++---
...le_NightlySnapshot.groovy => job_Release_NightlySnapshot.groovy} | 3 ++-
.test-infra/metrics/dashboards/stability_critical_jobs_status.json | 4 ++--
README.md | 6 +++---
14 files changed, 29 insertions(+), 26 deletions(-)
rename .test-infra/jenkins/{job_PostCommit_Go_GradleBuild.groovy =>
job_PostCommit_Go.groovy} (91%)
rename .test-infra/jenkins/{job_PostCommit_Java_GradleBuild.groovy =>
job_PostCommit_Java.groovy} (92%)
rename
.test-infra/jenkins/{job_PostCommit_Java_PortabilityApi_GradleBuild.groovy =>
job_PostCommit_Java_PortabilityApi.groovy} (94%)
rename .test-infra/jenkins/{job_Release_Gradle_NightlySnapshot.groovy =>
job_Release_NightlySnapshot.groovy} (96%)
[beam] 01/01: Merge pull request #7286 from swegner/jenkins_gradle
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit ec3f79214e9ef204fa32b744051a291fe4b61e23 Merge: 5ec695b 4f90294 Author: Scott Wegner AuthorDate: Fri Dec 14 13:24:35 2018 -0800 Merge pull request #7286 from swegner/jenkins_gradle Remove Gradle from Jenkins job names .github/PULL_REQUEST_TEMPLATE.md| 4 ++-- ...ob_PostCommit_Go_GradleBuild.groovy => job_PostCommit_Go.groovy} | 3 ++- ...ostCommit_Java_GradleBuild.groovy => job_PostCommit_Java.groovy} | 3 ++- ...GradleBuild.groovy => job_PostCommit_Java_PortabilityApi.groovy} | 3 ++- .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Apex.groovy | 5 ++--- .../jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy | 5 ++--- .../jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy| 3 ++- .../jenkins/job_PostCommit_Java_ValidatesRunner_Gearpump.groovy | 5 ++--- ...b_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy | 3 ++- .../jenkins/job_PostCommit_Java_ValidatesRunner_Samza.groovy| 3 ++- .../jenkins/job_PostCommit_Java_ValidatesRunner_Spark.groovy| 5 ++--- ...le_NightlySnapshot.groovy => job_Release_NightlySnapshot.groovy} | 3 ++- .test-infra/metrics/dashboards/stability_critical_jobs_status.json | 4 ++-- README.md | 6 +++--- 14 files changed, 29 insertions(+), 26 deletions(-)
[beam] branch asf-site updated: Publishing website 2018/12/14 20:15:30 at commit 5ec695b
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new de30525 Publishing website 2018/12/14 20:15:30 at commit 5ec695b de30525 is described below commit de305250739df1a016e75ab3b7518526659f1e42 Author: jenkins AuthorDate: Fri Dec 14 20:15:30 2018 + Publishing website 2018/12/14 20:15:30 at commit 5ec695b --- .../generated-content/contribute/ptransform-style-guide/index.html| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/website/generated-content/contribute/ptransform-style-guide/index.html b/website/generated-content/contribute/ptransform-style-guide/index.html index 275ad29..445300f 100644 --- a/website/generated-content/contribute/ptransform-style-guide/index.html +++ b/website/generated-content/contribute/ptransform-style-guide/index.html @@ -728,8 +728,8 @@ Strive to make such incompatible behavior changes cause a compile error (e.g. it Do: - If possible, just use PTransform composition as an extensibility device - i.e. if the same effect can be achieved by the user applying the transform in their pipeline and composing it with another PTransform, then the transform itself should not be extensible. E.g., a transform that writes JSON objects to a third-party system should take a PCollection(assuming it is possible to provide [...] - If extensibility by user code is necessary inside the transform, pass the user code as a SerializableFunction or define your own serializable function-like type (ideally single-method, for interoperability with Java 8 lambdas). Because Java erases the types of lambdas, you should be sure to have adequate type information even if a raw-type SerializableFunction is provided by the user. See PTransform, then the transform itself should not be extensible. E.g., a transform that writes JSON objects to a third-party system should take a PCollection (assuming it is possible to provide [...] + If extensibility by user code is necessary inside the transform, pass the user code as a ProcessFunction or define your own serializable function-like type (ideally single-method, for interoperability with Java 8 lambdas). Because Java erases the types of lambdas, you should be sure to have adequate type information even if a raw-type ProcessFunction is provided by the user. See
[beam] branch master updated: [BEAM-6150] Superinterface for SerializableFunction allowing declared exceptions (#7160)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 5ec695b [BEAM-6150] Superinterface for SerializableFunction allowing
declared exceptions (#7160)
5ec695b is described below
commit 5ec695b9397991996fb215a6f18f33e72d860e53
Author: Jeff Klukas
AuthorDate: Fri Dec 14 15:13:46 2018 -0500
[BEAM-6150] Superinterface for SerializableFunction allowing declared
exceptions (#7160)
* [BEAM-6150] Superinterface for SerializableFunction allowing declared
exceptions
Also provides an equivalent superclass for SimpleFunction.
See https://issues.apache.org/jira/browse/BEAM-6150
* Update MapElements docs to remove Java 7 references
* Update tests to use ProcessFunction
* Include tests for both existing and new classes
* Remove Java 7 reference in ptransform style guide
---
.../org/apache/beam/sdk/transforms/Contextful.java | 4 +-
.../org/apache/beam/sdk/transforms/Filter.java | 23 +--
.../beam/sdk/transforms/FlatMapElements.java | 40 +++--
...{SimpleFunction.java => InferableFunction.java} | 49 ---
.../apache/beam/sdk/transforms/MapElements.java| 36 ++---
...ializableFunction.java => ProcessFunction.java} | 17 ++-
.../beam/sdk/transforms/SerializableFunction.java | 11 +-
.../apache/beam/sdk/transforms/SimpleFunction.java | 38 +
.../org/apache/beam/sdk/transforms/ToString.java | 8 +-
.../apache/beam/sdk/values/TypeDescriptors.java| 40 ++---
.../org/apache/beam/sdk/transforms/FilterTest.java | 17 +++
.../beam/sdk/transforms/FlatMapElementsTest.java | 60 +++-
.../beam/sdk/transforms/MapElementsTest.java | 163 -
website/src/contribute/ptransform-style-guide.md | 4 +-
14 files changed, 358 insertions(+), 152 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Contextful.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Contextful.java
index 7e788cf..97a994f 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Contextful.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Contextful.java
@@ -104,11 +104,11 @@ public final class Contextful implements
Serializable {
}
/**
- * Wraps a {@link SerializableFunction} as a {@link Contextful} of {@link
Fn} with empty {@link
+ * Wraps a {@link ProcessFunction} as a {@link Contextful} of {@link Fn}
with empty {@link
* Requirements}.
*/
public static Contextful> fn(
- final SerializableFunction fn) {
+ final ProcessFunction fn) {
return new Contextful<>((element, c) -> fn.apply(element),
Requirements.empty());
}
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java
index 4bffeb6..aa9d2cd 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Filter.java
@@ -32,7 +32,7 @@ public class Filter extends PTransform,
PCollection> {
/**
* Returns a {@code PTransform} that takes an input {@code PCollection}
and returns a {@code
* PCollection} with elements that satisfy the given predicate. The
predicate must be a {@code
- * SerializableFunction}.
+ * ProcessFunction}.
*
* Example of use:
*
@@ -46,7 +46,7 @@ public class Filter extends PTransform,
PCollection> {
* #greaterThanEq}, which return elements satisfying various inequalities
with the specified value
* based on the elements' natural ordering.
*/
- public static >
Filter by(
+ public static > Filter
by(
PredicateT predicate) {
return new Filter<>(predicate);
}
@@ -71,7 +71,7 @@ public class Filter extends PTransform,
PCollection> {
* See also {@link #by}, which returns elements that satisfy the given
predicate.
*/
public static > Filter lessThan(final T value) {
-return by((SerializableFunction) input ->
input.compareTo(value) < 0)
+return by((ProcessFunction) input -> input.compareTo(value) <
0)
.described(String.format("x < %s", value));
}
@@ -95,7 +95,7 @@ public class Filter extends PTransform,
PCollection> {
* See also {@link #by}, which returns elements that satisfy the given
predicate.
*/
public static > Filter greaterThan(final T value)
{
-return by((SerializableFunction) input ->
input.compareTo(value) > 0)
+return by((ProcessFunction) input -> input.compareTo(value) >
0)
.described(String.format("x > %s", value));
}
@@ -119,7 +119,7 @@ public class Filter extends PTransform,
PCollection> {
* See also {@link #by}, which returns elements that satisfy the given
predicate.
*/
public static > F
[beam] branch master updated (977080f -> 71890da)
This is an automated email from the ASF dual-hosted git repository. goenka pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 977080f Merge pull request #7220: [BEAM-6191] Remove redundant error logging for Dataflow exception handling add f6c1dd5 [BEAM-6179] Fixing bundle estimation when all xs are same add 71890da Merge pull request #7280 from angoenka/fix_bundle_estimation No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/util.py | 8 +++- sdks/python/apache_beam/transforms/util_test.py | 8 2 files changed, 15 insertions(+), 1 deletion(-)
[beam] branch master updated (54e2fc1 -> 977080f)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 54e2fc1 [BEAM-6206] Add CustomHttpErrors a tool to allow adding custom errors for specific failing http calls. Plus, add a custom error message in BigQueryServicesImpl. (#7270) add 69358c5 [BEAM-6191] Remove redundant error logging for Dataflow exception handling new 977080f Merge pull request #7220: [BEAM-6191] Remove redundant error logging for Dataflow exception handling The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/dataflow/worker/WorkItemStatusClient.java | 9 ++--- .../dataflow/worker/util/common/worker/MapTaskExecutor.java | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-)
[beam] 01/01: Merge pull request #7220: [BEAM-6191] Remove redundant error logging for Dataflow exception handling
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 977080f908ac1a8d27d6aab3f4df5550062221ad Merge: 54e2fc1 69358c5 Author: Scott Wegner AuthorDate: Fri Dec 14 09:44:21 2018 -0800 Merge pull request #7220: [BEAM-6191] Remove redundant error logging for Dataflow exception handling .../beam/runners/dataflow/worker/WorkItemStatusClient.java | 9 ++--- .../dataflow/worker/util/common/worker/MapTaskExecutor.java | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-)
[beam] branch master updated: [BEAM-6206] Add CustomHttpErrors a tool to allow adding custom errors for specific failing http calls. Plus, add a custom error message in BigQueryServicesImpl. (#7270)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 54e2fc1 [BEAM-6206] Add CustomHttpErrors a tool to allow adding
custom errors for specific failing http calls. Plus, add a custom error message
in BigQueryServicesImpl. (#7270)
54e2fc1 is described below
commit 54e2fc12ad6c07c43782d03fd95241934b36bda6
Author: Alex Amato
AuthorDate: Fri Dec 14 09:30:48 2018 -0800
[BEAM-6206] Add CustomHttpErrors a tool to allow adding custom errors for
specific failing http calls. Plus, add a custom error message in
BigQueryServicesImpl. (#7270)
---
.../org/apache/beam/sdk/util/CustomHttpErrors.java | 141 +
.../apache/beam/sdk/util/HttpCallCustomError.java | 25
.../org/apache/beam/sdk/util/HttpCallMatcher.java | 28
.../apache/beam/sdk/util/HttpRequestWrapper.java | 40 ++
.../apache/beam/sdk/util/HttpResponseWrapper.java | 38 ++
.../beam/sdk/util/RetryHttpRequestInitializer.java | 32 -
.../apache/beam/sdk/util/CustomHttpErrorsTest.java | 128 +++
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 16 +++
8 files changed, 443 insertions(+), 5 deletions(-)
diff --git
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/CustomHttpErrors.java
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/CustomHttpErrors.java
new file mode 100644
index 000..db46d98
--- /dev/null
+++
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/CustomHttpErrors.java
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * An optional component to use with the {@code RetryHttpRequestInitializer}
in order to provide
+ * custom errors for failing http calls. This class allows you to specify
custom error messages
+ * which match specific error codes and containing strings in the URL. The
first matcher to match
+ * the request and response will be used to provide the custom error.
+ *
+ * The intended use case here is to examine one of the logs emitted by a
failing call made by the
+ * RetryHttpRequestInitializer, and then adding a custom error message which
matches the URL and
+ * code for it.
+ *
+ * Usage: See more in CustomHttpErrorsTest.
+ *
+ * {@code
+ * CustomHttpErrors.Builder builder = new CustomHttpErrors.Builder();
+ * builder.addErrorForCodeAndUrlContains(403,"/tables?", "Custom Error Msg");
+ * CustomHttpErrors customErrors = builder.build();
+ *
+ *
+ * RetryHttpRequestInitializer initializer = ...
+ * initializer.setCustomErrors(customErrors);
+ * }
+ *
+ * Suggestions for future enhancements to anyone upgrading this file:
+ *
+ *
+ * This class is left open for extension, to allow different functions
for HttpCallMatcher and
+ * HttpCallCustomError to match and log errors. For example, new
functionality may include
+ * matching an error based on the HttpResponse body. Additionally,
extracting and logging
+ * strings from the HttpResponse body may make useful functionality.
+ * Add a methods to add custom errors based on inspecting the contents
of the HttpRequest and
+ * HttpResponse
+ * Be sure to update the HttpRequestWrapper and HttpResponseWrapper with
any new getters that
+ * you may use. The wrappers were introduced to add a layer of
indirection which could be
+ * mocked mocked out in tests. This was unfortunately needed because
mockito cannot mock final
+ * classes and its non trivial to just construct HttpRequest and
HttpResponse objects.
+ * Making matchers composable with an AND operator may simplify
enhancing this code, if
+ * several different matchers are used.
+ *
+ *
+ *
+ */
+public class CustomHttpErrors {
+
+ /**
+ * A simple Tuple class for creating a list of HttpResponseMatcher and
HttpResponseCustomError to
+
[beam] branch master updated: Merge pull request #7267: [BEAM-4454] Support Avro POJO objects
This is an automated email from the ASF dual-hosted git repository.
reuvenlax pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 788ce61 Merge pull request #7267: [BEAM-4454] Support Avro POJO
objects
788ce61 is described below
commit 788ce61bd7c48bc16a8eaa93a46ac403155f4422
Author: reuvenlax
AuthorDate: Fri Dec 14 09:15:56 2018 -0800
Merge pull request #7267: [BEAM-4454] Support Avro POJO objects
* Add remaining Schema support for AVRO records:
* Add support for SpecificRecord using ByteBuddy codegen.
* Add helper methods for GenericRecord.
* Fix uncovered bugs involving nullable support.
---
...ificRecordSchema.java => AvroRecordSchema.java} | 19 +-
.../schemas/AvroSpecificRecordGetterFactory.java | 30 ---
.../AvroSpecificRecordUserTypeCreatorFactory.java | 29 ---
.../sdk/schemas/FieldValueTypeInformation.java | 127 +---
.../beam/sdk/schemas/FromRowUsingCreator.java | 2 +-
.../sdk/schemas/GetterBasedSchemaProvider.java | 2 -
.../apache/beam/sdk/schemas/JavaBeanSchema.java| 68 ++-
.../apache/beam/sdk/schemas/JavaFieldSchema.java | 37 +++-
.../schemas/PojoTypeUserTypeCreatorFactory.java| 28 ---
.../schemas/SchemaUserTypeConstructorCreator.java | 2 +-
.../AvroSpecificRecordTypeInformationFactory.java | 32 ---
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 123 +++
...ionFactory.java => FieldValueTypeSupplier.java} | 18 +-
.../sdk/schemas/utils/JavaBeanGetterFactory.java | 32 ---
.../sdk/schemas/utils/JavaBeanSetterFactory.java | 31 ---
.../utils/JavaBeanTypeInformationFactory.java | 32 ---
.../beam/sdk/schemas/utils/JavaBeanUtils.java | 162 +--
.../apache/beam/sdk/schemas/utils/POJOUtils.java | 101 -
.../sdk/schemas/utils/PojoValueGetterFactory.java | 31 ---
.../sdk/schemas/utils/PojoValueSetterFactory.java | 31 ---
.../beam/sdk/schemas/utils/ReflectUtils.java | 13 +-
.../sdk/schemas/utils/StaticSchemaInference.java | 104 +-
.../apache/beam/sdk/schemas/AvroSchemaTest.java| 226 -
.../beam/sdk/schemas/utils/JavaBeanUtilsTest.java | 49 ++---
.../beam/sdk/schemas/utils/POJOUtilsTest.java | 16 +-
25 files changed, 652 insertions(+), 693 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroSpecificRecordSchema.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java
similarity index 67%
rename from
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroSpecificRecordSchema.java
rename to
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java
index d8e4bda..29bf51a 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroSpecificRecordSchema.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroRecordSchema.java
@@ -17,35 +17,34 @@
*/
package org.apache.beam.sdk.schemas;
-import org.apache.avro.specific.SpecificRecord;
-import
org.apache.beam.sdk.schemas.utils.AvroSpecificRecordTypeInformationFactory;
import org.apache.beam.sdk.schemas.utils.AvroUtils;
import org.apache.beam.sdk.values.TypeDescriptor;
/**
- * A {@link SchemaProvider} for AVRO generated SpecificRecords.
+ * A {@link SchemaProvider} for AVRO generated SpecificRecords and POJOs.
*
- * This provider infers a schema from generates SpecificRecord objects, and
creates schemas and
- * rows that bind to the appropriate fields.
+ * This provider infers a schema from generated SpecificRecord objects, and
creates schemas and
+ * rows that bind to the appropriate fields. This provider also infers schemas
from Java POJO
+ * objects, creating a schema that matches that inferred by the AVRO libraries.
*/
-public class AvroSpecificRecordSchema extends GetterBasedSchemaProvider {
+public class AvroRecordSchema extends GetterBasedSchemaProvider {
@Override
public Schema schemaFor(TypeDescriptor typeDescriptor) {
-return AvroUtils.getSchema((Class)
typeDescriptor.getRawType());
+return AvroUtils.getSchema(typeDescriptor.getRawType());
}
@Override
public FieldValueGetterFactory fieldValueGetterFactory() {
-return new AvroSpecificRecordGetterFactory();
+return AvroUtils::getGetters;
}
@Override
public UserTypeCreatorFactory schemaTypeCreatorFactory() {
-return new AvroSpecificRecordUserTypeCreatorFactory();
+return AvroUtils::getCreator;
}
@Override
public FieldValueTypeInformationFactory fieldValueTypeInformationFactory() {
-return new AvroSpecificRecordTypeInformationFactory();
+return AvroUtils::getFieldTypes;
}
}
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroSpecificRecordGetterFactory.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AvroSpecificRecordGett
[beam] branch master updated (88f181c -> 2886473)
This is an automated email from the ASF dual-hosted git repository. thw pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 88f181c Merge pull request #7283: [BEAM-6229] Fix LoadTestResult to store propoer timestamp and runtime add fc38359 [BEAM-6227] Fix GroupByKey with null values in Flink Runner new 2886473 Merge pull request #7282: [BEAM-6227] Fix GroupByKey with null values in Flink Runner The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../streaming/state/FlinkStateInternals.java | 58 +++--- .../flink/streaming/GroupByWithNullValuesTest.java | 92 ++ 2 files changed, 138 insertions(+), 12 deletions(-) create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/GroupByWithNullValuesTest.java
[beam] 01/01: Merge pull request #7282: [BEAM-6227] Fix GroupByKey with null values in Flink Runner
This is an automated email from the ASF dual-hosted git repository. thw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 28864734cf108a49fa5751aa1f867e97c0e069d0 Merge: 88f181c fc38359 Author: Thomas Weise AuthorDate: Fri Dec 14 08:41:38 2018 -0800 Merge pull request #7282: [BEAM-6227] Fix GroupByKey with null values in Flink Runner .../streaming/state/FlinkStateInternals.java | 58 +++--- .../flink/streaming/GroupByWithNullValuesTest.java | 92 ++ 2 files changed, 138 insertions(+), 12 deletions(-)
[beam] 01/01: Merge pull request #7283: [BEAM-6229] Fix LoadTestResult to store propoer timestamp and runtime
This is an automated email from the ASF dual-hosted git repository. lgajowy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 88f181c6108f9de39e552a37bc74dda60d3075bb Merge: a34e459 0d60a89 Author: Łukasz Gajowy AuthorDate: Fri Dec 14 16:24:49 2018 +0100 Merge pull request #7283: [BEAM-6229] Fix LoadTestResult to store propoer timestamp and runtime .../org/apache/beam/sdk/loadtests/ConsoleResultPublisher.java | 2 +- .../main/java/org/apache/beam/sdk/loadtests/LoadTestResult.java | 8 2 files changed, 5 insertions(+), 5 deletions(-)
[beam] branch master updated (a34e459 -> 88f181c)
This is an automated email from the ASF dual-hosted git repository. lgajowy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a34e459 Merge pull request #7274 Add reshuffle option to Create. add 0d60a89 [BEAM-6229] Fix LoadTestResult to store propoer timestamp and runtime new 88f181c Merge pull request #7283: [BEAM-6229] Fix LoadTestResult to store propoer timestamp and runtime The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/sdk/loadtests/ConsoleResultPublisher.java | 2 +- .../main/java/org/apache/beam/sdk/loadtests/LoadTestResult.java | 8 2 files changed, 5 insertions(+), 5 deletions(-)
[beam] branch master updated (e21b80a -> a34e459)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e21b80a Merge pull request #7278: Updates blog for 2.9.0 release add eae5521 Add reshuffle option to Create. new a34e459 Merge pull request #7274 Add reshuffle option to Create. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/transforms/core.py | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
[beam] 01/01: Merge pull request #7274 Add reshuffle option to Create.
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a34e45981818fb63e03d6d1e3d0438b7863fed50 Merge: e21b80a eae5521 Author: Robert Bradshaw AuthorDate: Fri Dec 14 10:02:32 2018 +0100 Merge pull request #7274 Add reshuffle option to Create. Add reshuffle option to Create. sdks/python/apache_beam/transforms/core.py | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
