[beam] branch master updated: [BEAM-6504] Create Portable sideInput handler for Dataflow (Part One) (#7619)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 7f19997 [BEAM-6504] Create Portable sideInput handler for Dataflow
(Part One) (#7619)
7f19997 is described below
commit 7f199977b97eca55d6aea6290c2a8bf7a95d7812
Author: Ruoyun Huang
AuthorDate: Fri Feb 15 12:54:13 2019 -0800
[BEAM-6504] Create Portable sideInput handler for Dataflow (Part One)
(#7619)
---
.../control/DataflowSideInputHandlerFactory.java | 151
.../DataflowSideInputHandlerFactoryTest.java | 193 +
2 files changed, 344 insertions(+)
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactory.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactory.java
new file mode 100644
index 000..df02c46
--- /dev/null
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/DataflowSideInputHandlerFactory.java
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.fn.control;
+
+import static
org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkArgument;
+import static
org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.SideInputReader;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandlers;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.transforms.Materializations;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.values.PCollectionView;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** This handles sideinput in Dataflow. The caller should be based on
ExecutableStage framework. */
+public class DataflowSideInputHandlerFactory
+implements StateRequestHandlers.SideInputHandlerFactory {
+ private static final Logger LOG =
LoggerFactory.getLogger(DataflowSideInputHandlerFactory.class);
+
+ private final Map ptransformIdToSideInputReader;
+ private final Map>
+ sideInputIdToPCollectionViewMap;
+
+ static DataflowSideInputHandlerFactory of(
+ Map ptransformIdToSideInputReader,
+ Map>
+ sideInputIdToPCollectionViewMap) {
+return new DataflowSideInputHandlerFactory(
+ptransformIdToSideInputReader, sideInputIdToPCollectionViewMap);
+ }
+
+ private DataflowSideInputHandlerFactory(
+ Map ptransformIdToSideInputReader,
+ Map>
+ sideInputIdToPCollectionViewMap) {
+this.ptransformIdToSideInputReader = ptransformIdToSideInputReader;
+this.sideInputIdToPCollectionViewMap = sideInputIdToPCollectionViewMap;
+ }
+
+ @Override
+ public
StateRequestHandlers.SideInputHandler forSideInput(
+ String pTransformId,
+ String sideInputId,
+ RunnerApi.FunctionSpec accessPattern,
+ Coder elementCoder,
+ Coder windowCoder) {
+checkArgument(
+pTransformId != null && pTransformId.length() > 0, "Expect a valid
PTransform ID.");
+
+SideInputReader sideInputReader =
ptransformIdToSideInputReader.get(pTransformId);
+checkState(sideInputReader != null, String.format("Unknown PTransform
'%s'", pTransformId));
+
+PCollectionView> view =
+(PCollectionView>)
+sideInputIdToPCollectionViewMap.get(
+RunnerApi.ExecutableStagePayload.SideInputId.newBuilder()
+.setTransformId(pTransformId)
+.setLocalName(sideInputId)
+.build());
+checkState(
+view != null,
+String.format("Unknown sid
[beam] branch master updated: Propagate UserStateReferences to ExecutableStagePayload
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 19dbd8c Propagate UserStateReferences to ExecutableStagePayload
new a103eda Merge pull request #7844: [BEAM-6672] Propagate
UserStateReferences to ExecutableStagePayload
19dbd8c is described below
commit 19dbd8cc587b60be6607cc2a09428400049f0b4a
Author: Sam Rohde
AuthorDate: Thu Feb 14 11:05:18 2019 -0800
Propagate UserStateReferences to ExecutableStagePayload
---
.../graph/CreateExecutableStageNodeFunction.java | 19 ++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateExecutableStageNodeFunction.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateExecutableStageNodeFunction.java
index 8984d70..a2cf3ab 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateExecutableStageNodeFunction.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/graph/CreateExecutableStageNodeFunction.java
@@ -40,6 +40,7 @@ import java.util.function.Function;
import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.UserStateId;
import org.apache.beam.model.pipeline.v1.RunnerApi.StandardPTransforms;
import org.apache.beam.runners.core.construction.*;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
@@ -280,6 +281,7 @@ public class CreateExecutableStageNodeFunction
componentsBuilder.putAllCoders(sdkComponents.toComponents().getCodersMap());
Set executableStageTransforms = new HashSet<>();
Set executableStageTimers = new HashSet<>();
+List userStateIds = new ArrayList<>();
for (ParallelInstructionNode node :
Iterables.filter(input.nodes(), ParallelInstructionNode.class)) {
@@ -332,11 +334,18 @@ public class CreateExecutableStageNodeFunction
}
// Build the necessary components to inform the SDK Harness of the
pipeline's
-// timers.
+// user timers and user state.
for (Map.Entry entry :
parDoPayload.getTimerSpecsMap().entrySet()) {
timerIds.add(entry.getKey());
}
+for (Map.Entry entry :
+parDoPayload.getStateSpecsMap().entrySet()) {
+ UserStateId.Builder builder = UserStateId.newBuilder();
+ builder.setTransformId(parDoPTransformId);
+ builder.setLocalName(entry.getKey());
+ userStateIds.add(builder.build());
+}
transformSpec
.setUrn(PTransformTranslation.PAR_DO_TRANSFORM_URN)
@@ -373,6 +382,8 @@ public class CreateExecutableStageNodeFunction
String.format("Unknown type of ParallelInstruction %s",
parallelInstruction));
}
+ // Even though this is a for-loop, there is only going to be a single
PCollection as the
+ // predecessor in a ParDo. This PCollection is called the "main input".
for (Node predecessorOutput : input.predecessors(node)) {
pTransform.putInputs(
"generatedInput" + idGenerator.getId(),
nodesToPCollections.get(predecessorOutput));
@@ -410,6 +421,12 @@ public class CreateExecutableStageNodeFunction
Set executableStageSideInputs = new HashSet<>();
Set executableStageUserStateReference = new
HashSet<>();
+
+for (UserStateId userStateId : userStateIds) {
+ executableStageUserStateReference.add(
+ UserStateReference.fromUserStateId(userStateId,
executableStageComponents));
+}
+
ExecutableStage executableStage =
ImmutableExecutableStage.ofFullComponents(
executableStageComponents,
[beam] branch master updated (dbe86f8 -> 6ce9701)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from dbe86f8 [BEAM-6583] Update minimal Python 3 requirements. (#7839) new cadb6f7 [BEAM-6630] upgrade to gradle 5.2 new b8c247c [BEAM-6635] increase maxHeapSize for test tasks new 6ce9701 Merge pull request #7787: [BEAM-6630] upgrade to gradle 5.2 The 20160 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: build.gradle | 2 +- buildSrc/build.gradle | 30 +++--- .../org/apache/beam/gradle/BeamModulePlugin.groovy | 8 -- gradle/wrapper/gradle-wrapper.properties | 2 +- 4 files changed, 23 insertions(+), 19 deletions(-)
[beam] branch master updated: Fix grammar for Flatten in section 4.2.5
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2acfa71 Fix grammar for Flatten in section 4.2.5
new 6d6385f Merge pull request #7833: Fix grammar for Flatten in section
4.2.5 of Programming Guide
2acfa71 is described below
commit 2acfa7189490ecf50893cdcce466df3bd656452e
Author: ttanay
AuthorDate: Thu Feb 14 02:22:35 2019 +0530
Fix grammar for Flatten in section 4.2.5
The section about Flatten had "`Flatten` and is a Beam transform...".
It should be "`Flatten` is a Beam transform...".
---
website/src/documentation/programming-guide.md | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/website/src/documentation/programming-guide.md
b/website/src/documentation/programming-guide.md
index 45fb6ee..a322001 100644
--- a/website/src/documentation/programming-guide.md
+++ b/website/src/documentation/programming-guide.md
@@ -1194,7 +1194,7 @@ player_accuracies = ...
4.2.5. Flatten {#flatten}
[`Flatten`](https://beam.apache.org/releases/javadoc/{{
site.release_latest
}}/index.html?org/apache/beam/sdk/transforms/Flatten.html)
-[`Flatten`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py)
and
+[`Flatten`](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/core.py)
is a Beam transform for `PCollection` objects that store the same data type.
`Flatten` merges multiple `PCollection` objects into a single logical
`PCollection`.
@@ -1524,7 +1524,7 @@ together.
4.5.2. Emitting to multiple outputs in your DoFn {#multiple-outputs-dofn}
```java
-// Inside your ParDo's DoFn, you can emit an element to a specific output
PCollection by providing a
+// Inside your ParDo's DoFn, you can emit an element to a specific output
PCollection by providing a
// MultiOutputReceiver to your process method, and passing in the appropriate
TupleTag to obtain an OutputReceiver.
// After your ParDo, extract the resulting output PCollections from the
returned PCollectionTuple.
// Based on the previous example, this shows the DoFn emitting to the main
output and two additional outputs.
@@ -1616,7 +1616,7 @@ The `PipelineOptions` for the current pipeline can always
be accessed in a proce
`@OnTimer` methods can also access many of these parameters. Timestamp,
window, `PipelineOptions`, `OutputReceiver`, and
`MultiOutputReceiver` parameters can all be accessed in an `@OnTimer` method.
In addition, an `@OnTimer` method can take
a parameter of type `TimeDomain` which tells whether the timer is based on
event time or processing time.
-Timers are explained in more detail in the
+Timers are explained in more detail in the
[Timely (and Stateful) Processing with Apache Beam]({{ site.baseurl
}}/blog/2017/08/28/timely-processing.html) blog post.
### 4.6. Composite transforms {#composite-transforms}
[beam] branch master updated: Switching google_api_services_dataflow to v1b3-rev20190126 (#7828)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3cc8d36 Switching google_api_services_dataflow to v1b3-rev20190126
(#7828)
3cc8d36 is described below
commit 3cc8d36add9f8aa478f265f6cf0cccfc266e77a9
Author: drieber
AuthorDate: Wed Feb 13 11:04:30 2019 -0800
Switching google_api_services_dataflow to v1b3-rev20190126 (#7828)
---
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 4c58acf..a0abc22 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -412,7 +412,7 @@ class BeamModulePlugin implements Plugin {
google_api_services_bigquery:
"com.google.apis:google-api-services-bigquery:v2-rev20181104-$google_clients_version",
google_api_services_clouddebugger :
"com.google.apis:google-api-services-clouddebugger:v2-rev20180801-$google_clients_version",
google_api_services_cloudresourcemanager:
"com.google.apis:google-api-services-cloudresourcemanager:v1-rev20181015-$google_clients_version",
-google_api_services_dataflow:
"com.google.apis:google-api-services-dataflow:v1b3-rev20181107-$google_clients_version",
+google_api_services_dataflow:
"com.google.apis:google-api-services-dataflow:v1b3-rev20190126-$google_clients_version",
google_api_services_pubsub :
"com.google.apis:google-api-services-pubsub:v1-rev20181105-$google_clients_version",
google_api_services_storage :
"com.google.apis:google-api-services-storage:v1-rev20181013-$google_clients_version",
google_auth_library_credentials :
"com.google.auth:google-auth-library-credentials:$google_auth_version",
[beam] branch master updated: [BEAM-6653] Implement Lullz logging for the Java SDK (#7818)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2e8fc2e [BEAM-6653] Implement Lullz logging for the Java SDK (#7818)
2e8fc2e is described below
commit 2e8fc2e5f7d48e29fd8e9abdf41fcc100deced50
Author: Alex Amato
AuthorDate: Tue Feb 12 13:07:18 2019 -0800
[BEAM-6653] Implement Lullz logging for the Java SDK (#7818)
---
.../runners/core/metrics/SimpleExecutionState.java | 70 --
.../runners/core/metrics/SimpleStateRegistry.java | 2 +-
.../core/metrics/SimpleExecutionStateTest.java | 33 +-
.../core/metrics/SimpleStateRegistryTest.java | 12 +++-
.../fn/harness/control/ProcessBundleHandler.java | 9 +--
.../harness/data/PCollectionConsumerRegistry.java | 4 +-
.../harness/data/PTransformFunctionRegistry.java | 18 --
7 files changed, 124 insertions(+), 24 deletions(-)
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java
index 88f742c..9ae6741 100644
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java
@@ -21,6 +21,12 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import
org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState;
+import
org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting;
+import org.joda.time.Duration;
+import org.joda.time.format.PeriodFormatter;
+import org.joda.time.format.PeriodFormatterBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* Simple state class which collects the totalMillis spent in the state.
Allows storing an arbitrary
@@ -30,14 +36,41 @@ import
org.apache.beam.runners.core.metrics.ExecutionStateTracker.ExecutionState
public class SimpleExecutionState extends ExecutionState {
private long totalMillis = 0;
private HashMap labelsMetadata;
+ private String urn;
+
+ private static final Logger LOG =
LoggerFactory.getLogger(SimpleExecutionState.class);
+
+ private static final PeriodFormatter DURATION_FORMATTER =
+ new PeriodFormatterBuilder()
+ .appendDays()
+ .appendSuffix("d")
+ .minimumPrintedDigits(2)
+ .appendHours()
+ .appendSuffix("h")
+ .printZeroAlways()
+ .appendMinutes()
+ .appendSuffix("m")
+ .appendSeconds()
+ .appendSuffix("s")
+ .toFormatter();
/**
- * @param urn A string urn for the execution time metric.
+ * @param stateName A state name to be used in lull logging when stuck in a
state.
+ * @param urn A optional string urn for an execution time metric.
* @param labelsMetadata arbitrary metadata to use for reporting purposes.
*/
- public SimpleExecutionState(String urn, HashMap
labelsMetadata) {
-super(urn);
+ public SimpleExecutionState(
+ String stateName, String urn, HashMap labelsMetadata) {
+super(stateName);
+this.urn = urn;
this.labelsMetadata = labelsMetadata;
+if (this.labelsMetadata == null) {
+ this.labelsMetadata = new HashMap();
+}
+ }
+
+ public String getUrn() {
+return this.urn;
}
public Map getLabels() {
@@ -53,8 +86,37 @@ public class SimpleExecutionState extends ExecutionState {
return totalMillis;
}
+ @VisibleForTesting
+ public String getLullMessage(Thread trackedThread, Duration millis) {
+// TODO(ajamato): Share getLullMessage code with DataflowExecutionState.
+String userStepName =
+
this.labelsMetadata.getOrDefault(SimpleMonitoringInfoBuilder.PTRANSFORM_LABEL,
null);
+StringBuilder message = new StringBuilder();
+message.append("Processing stuck");
+if (userStepName != null) {
+ message.append(" in step ").append(userStepName);
+}
+message
+.append(" for at least ")
+.append(formatDuration(millis))
+.append(" without outputting or completing in state ")
+.append(getStateName());
+message.append("\n");
+
+StackTraceElement[] fullTrace = trackedThread.getStackTrace();
+for (StackTraceElement e : fullTrace) {
+ message.append(" at ").append(e).append("\n");
+}
+return message.toString();
+ }
+
@Override
public void reportLull(Thread trackedThread, long millis) {
-// TOOD(ajamato): Implement lullz detection to log stuck PTransforms.
+LOG.warn(getLullMessage(trackedThread, Duration.millis(millis)));
+ }
+
+ @VisibleForTesting
+ static String format
[beam] branch master updated: [BEAM-5449] Adding Groovy file for ULR VR PostCommit. (#7680)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 139d29a [BEAM-5449] Adding Groovy file for ULR VR PostCommit. (#7680)
139d29a is described below
commit 139d29ac0211baa92da1bc9298faf8fc00e5c59e
Author: Daniel Oliveira
AuthorDate: Mon Feb 11 10:17:22 2019 -0800
[BEAM-5449] Adding Groovy file for ULR VR PostCommit. (#7680)
---
...t_Java_PortableValidatesRunner_Reference.groovy | 47 ++
1 file changed, 47 insertions(+)
diff --git
a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Reference.groovy
b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Reference.groovy
new file mode 100644
index 000..bd3f3b4
--- /dev/null
+++
b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Reference.groovy
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of ValidatesRunner tests against the Java Reference
+// Runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_PVR_Reference',
+'Run Java Reference Runner PortableValidatesRunner',
+'Java Reference Runner PortableValidatesRunner Tests',
+this) {
+ description(
+ 'Runs the Java PortableValidatesRunner suite on the Reference Runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Publish all test results to Jenkins
+ publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+ }
+
+ // Gradle goals for this job.
+ steps {
+gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':beam-runners-direct-java:validatesPortableRunner')
+ commonJobProperties.setGradleSwitches(delegate)
+}
+ }
+}
[beam] branch master updated (197a068 -> fdbc0b1)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 197a068 Merge pull request #7800: [BEAM-2939] Do not drop non-empty residual roots on bundle completion new c98685e Update GCP dependency versions. new ba414c7 Update non-vendored gRPC and Netty versions. new fdbc0b1 Merge pull request #7783: [BEAM-6628] Update GCP dependency versions. The 20056 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 26 +++--- .../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 14 +++- .../beam/sdk/io/gcp/spanner/SpannerReadIT.java | 6 ++--- .../beam/sdk/io/gcp/spanner/SpannerWriteIT.java| 6 ++--- 4 files changed, 32 insertions(+), 20 deletions(-)
[beam] branch master updated: upgrade to gogradle plugin 0.11.2
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 02be861 upgrade to gogradle plugin 0.11.2
new 622c9c8 Merge pull request #7762: upgrade to gogradle plugin 0.11.2
02be861 is described below
commit 02be861791ffbb7f642cfd8cd541db297f4057e5
Author: Michael Luckey <25622840+adude3...@users.noreply.github.com>
AuthorDate: Thu Feb 7 00:28:08 2019 +0100
upgrade to gogradle plugin 0.11.2
---
build.gradle | 4 ++--
.../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 6 ++
sdks/go/build.gradle | 4 ++--
sdks/go/container/build.gradle | 6 +++---
sdks/go/examples/build.gradle | 4 ++--
sdks/go/test/build.gradle | 4 ++--
sdks/java/container/build.gradle | 6 +++---
sdks/python/container/build.gradle | 10 +-
8 files changed, 25 insertions(+), 19 deletions(-)
diff --git a/build.gradle b/build.gradle
index 2af3d20..4a8a948 100644
--- a/build.gradle
+++ b/build.gradle
@@ -55,7 +55,7 @@ buildscript {
classpath "gradle.plugin.org.nosphere.apache:creadur-rat-gradle:0.3.1"
// Enable Apache license enforcement
classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:0.11.0"
// Enable Avro code generation
classpath "com.diffplug.spotless:spotless-plugin-gradle:3.17.0"
// Enable a code formatting plugin
-classpath "gradle.plugin.com.github.blindpirate:gogradle:0.10"
// Enable Go code compilation
+classpath "gradle.plugin.com.github.blindpirate:gogradle:0.11.2"
// Enable Go code compilation
classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.20.1"
// Enable building Docker containers
classpath "gradle.plugin.com.dorongold.plugins:task-tree:1.3.1"
// Adds a 'taskTree' task to print task dependency tree
classpath "com.github.jengelman.gradle.plugins:shadow:4.0.3"
// Enable shading Java dependencies
@@ -211,7 +211,7 @@ task javaPostCommitPortabilityApi () {
}
task goPreCommit() {
- dependsOn ":beam-sdks-go:test"
+ dependsOn ":beam-sdks-go:goTest"
dependsOn ":beam-sdks-go-examples:build"
dependsOn ":beam-sdks-go-test:build"
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index cb41298..47f97e1 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -1270,6 +1270,9 @@ class BeamModulePlugin implements Plugin {
/**
***/
project.ext.applyGoNature = {
+ // Define common lifecycle tasks and artifact types
+ project.apply plugin: 'base'
+
project.apply plugin: "com.github.blindpirate.gogradle"
project.golang { goVersion = '1.10' }
@@ -1283,6 +1286,9 @@ class BeamModulePlugin implements Plugin {
root 'github.com/apache/thrift'
emptyDir()
}
+project.clean.dependsOn project.goClean
+project.check.dependsOn project.goCheck
+project.assemble.dependsOn project.goBuild
}
project.idea {
diff --git a/sdks/go/build.gradle b/sdks/go/build.gradle
index fe117bb..5bc2e73 100644
--- a/sdks/go/build.gradle
+++ b/sdks/go/build.gradle
@@ -24,7 +24,7 @@ description = "Apache Beam :: SDKs :: Go"
golang {
packagePath = 'github.com/apache/beam/sdks/go'
- build {
+ goBuild {
// The symlinks makes it hard (impossible?) to do a wildcard build
// of pkg. Go build refuses to follow symlinks. Drop for now. The files
// are built when tested anyway.
@@ -37,7 +37,7 @@ golang {
}
// Ignore spurious vet errors during check for [BEAM-4831].
- vet {
+ goVet {
continueOnFailure = true
}
}
diff --git a/sdks/go/container/build.gradle b/sdks/go/container/build.gradle
index 24e7a99..276bde2 100644
--- a/sdks/go/container/build.gradle
+++ b/sdks/go/container/build.gradle
@@ -25,7 +25,7 @@ description = "Apache Beam :: SDKs :: Go :: Container"
// Figure out why the golang plugin doe
[beam] branch master updated: [BEAM-5303] Adding Python VR tests for Java Reference Runner.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3c5f3af [BEAM-5303] Adding Python VR tests for Java Reference Runner.
new 298c9bc Merge pull request #7768: [BEAM-5303] Adding Python VR tests
for Java Reference Runner.
3c5f3af is described below
commit 3c5f3afc7f8ee447d827a684d4c0fd7db02b6e29
Author: Daniel Oliveira
AuthorDate: Wed Feb 6 16:01:18 2019 -0800
[BEAM-5303] Adding Python VR tests for Java Reference Runner.
Adding a gradle target and a test class for running ValidatesRunner
tests with the Java Reference Runner. Also had to add a dependency to
base_image_requirements.txt to get it to run properly. Note that this
was also causing an error when running Flink's VR tests with Docker.
---
.../portable/job/ReferenceRunnerJobService.java| 37 -
.../portability/java_reference_runner_test.py | 163 +
sdks/python/build.gradle | 14 ++
sdks/python/container/base_image_requirements.txt | 3 +
4 files changed, 212 insertions(+), 5 deletions(-)
diff --git
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobService.java
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobService.java
index f0a83e1..07517af 100644
---
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobService.java
+++
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobService.java
@@ -43,6 +43,7 @@ import org.apache.beam.runners.fnexecution.FnService;
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.ServerFactory;
import
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactStagingService;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status;
import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException;
import org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.StreamObserver;
@@ -211,11 +212,18 @@ public class ReferenceRunnerJobService extends
JobServiceImplBase implements FnS
public void getState(
GetJobStateRequest request, StreamObserver
responseObserver) {
LOG.trace("{} {}", GetJobStateRequest.class.getSimpleName(), request);
-responseObserver.onNext(
-GetJobStateResponse.newBuilder()
-.setState(jobStates.getOrDefault(request.getJobId(),
Enum.UNRECOGNIZED))
-.build());
-responseObserver.onCompleted();
+try {
+ responseObserver.onNext(
+ GetJobStateResponse.newBuilder()
+ .setState(jobStates.getOrDefault(request.getJobId(),
Enum.UNRECOGNIZED))
+ .build());
+ responseObserver.onCompleted();
+} catch (Exception e) {
+ String errMessage =
+ String.format("Encountered Unexpected Exception for Invocation %s",
request.getJobId());
+ LOG.error(errMessage, e);
+ responseObserver.onError(Status.INTERNAL.withCause(e).asException());
+}
}
@Override
@@ -242,6 +250,25 @@ public class ReferenceRunnerJobService extends
JobServiceImplBase implements FnS
}
@Override
+ public void describePipelineOptions(
+ JobApi.DescribePipelineOptionsRequest request,
+ StreamObserver responseObserver)
{
+LOG.trace("{} {}",
JobApi.DescribePipelineOptionsRequest.class.getSimpleName(), request);
+try {
+ JobApi.DescribePipelineOptionsResponse response =
+ JobApi.DescribePipelineOptionsResponse.newBuilder()
+ .addAllOptions(
+
PipelineOptionsFactory.describe(PipelineOptionsFactory.getRegisteredOptions()))
+ .build();
+ responseObserver.onNext(response);
+ responseObserver.onCompleted();
+} catch (Exception e) {
+ LOG.error("Error describing pipeline options", e);
+ responseObserver.onError(Status.INTERNAL.withCause(e).asException());
+}
+ }
+
+ @Override
public void getMessageStream(
JobMessagesRequest request, StreamObserver
responseObserver) {
// Not implemented
diff --git
a/sdks/python/apache_beam/runners/portability/java_reference_runner_test.py
b/sdks/python/apache_beam/runners/portability/java_reference_runner_test.py
new file mode 100644
index 000..59cc5fb
--- /dev/null
+++ b/sdks/python/apache_beam/runners/portability/java_reference_runner_test.py
@@ -0,0 +1,163 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You
[beam] branch master updated (9b27a6c -> dabfada)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9b27a6c Merge pull request #7755: [BEAM-6615] OutputReceiver is not an annotation new 3606f9f Update README.md new defd239 Update .test-infra/dockerized-jenkins/README.md new dabfada Merge pull request #7769: Minor tweaks to Dockerized Jenkins README The 19991 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .test-infra/dockerized-jenkins/README.md | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-)
[beam] branch master updated: OutputReceiver is not an annotation
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e99a096 OutputReceiver is not an annotation
new 9b27a6c Merge pull request #7755: [BEAM-6615] OutputReceiver is not
an annotation
e99a096 is described below
commit e99a09649481dbf8a5114ecef3b3b9bb9aef56d5
Author: Brian Hulette
AuthorDate: Wed Feb 6 10:52:12 2019 -0800
OutputReceiver is not an annotation
---
.../java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index 3e86c79..e7ef933 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -125,7 +125,7 @@ import
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableLis
* lines.apply(ParDo.of(new DoFn() {
*{@literal @}ProcessElement
* public void processElement({@literal @}Element String line,
- * {@literal @}OutputReceiver r) {
+ * OutputReceiver r) {
* for (String word : line.split("[^a-zA-Z']+")) {
* r.output(word);
* }
@@ -134,7 +134,7 @@ import
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.ImmutableLis
* words.apply(ParDo.of(new DoFn() {
*{@literal @}ProcessElement
* public void processElement({@literal @}Element String word,
- * {@literal @}OutputReceiver r) {
+ * OutputReceiver r) {
* Integer length = word.length();
* r.output(length);
* }}));
[beam] branch master updated (c3b0f45 -> 696cf22)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c3b0f45 Merge pull request #7699: [BEAM-1132] add jacoco report on javaPreCommit new 2aa8a9f [BEAM-6431] Implement Execution Time metrics start,process,finish in the Java SDK new 4a785ba spotless and others new 887cf8a spotless new 0479831 added some test new 67bc9a4 Add more tests and the SimpleStateRegistry new eb6b4dd Iterated on PR new 696cf22 Merge pull request #7676: [BEAM-6431] Implement Execution Time metrics start,process,finish in the Java SDK The 19927 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../runners/core/metrics/SimpleExecutionState.java | 60 + .../core/metrics/SimpleMonitoringInfoBuilder.java | 14 +++- .../runners/core/metrics/SimpleStateRegistry.java | 49 +++ .../core/metrics/MonitoringInfoMatchers.java | 98 ++ .../core/metrics/SimpleExecutionStateTest.java | 56 + .../core/metrics/SimpleStateRegistryTest.java | 78 + runners/java-fn-execution/build.gradle | 1 + .../fnexecution/control/RemoteExecutionTest.java | 75 - .../java/org/apache/beam/fn/harness/FnHarness.java | 1 + .../fn/harness/control/ProcessBundleHandler.java | 40 ++--- .../harness/data/PCollectionConsumerRegistry.java | 34 +++- .../harness/data/PTransformFunctionRegistry.java | 49 --- .../beam/fn/harness/AssignWindowsRunnerTest.java | 5 +- .../beam/fn/harness/BeamFnDataReadRunnerTest.java | 11 ++- .../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 12 ++- .../beam/fn/harness/BoundedSourceRunnerTest.java | 11 ++- .../apache/beam/fn/harness/CombineRunnersTest.java | 38 ++--- .../apache/beam/fn/harness/FlattenRunnerTest.java | 8 +- .../beam/fn/harness/FnApiDoFnRunnerTest.java | 47 +++ .../apache/beam/fn/harness/MapFnRunnersTest.java | 32 --- .../data/PCollectionConsumerRegistryTest.java | 10 ++- .../data/PTransformFunctionRegistryTest.java | 8 +- 22 files changed, 631 insertions(+), 106 deletions(-) create mode 100644 runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleExecutionState.java create mode 100644 runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleStateRegistry.java create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoMatchers.java create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SimpleExecutionStateTest.java create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SimpleStateRegistryTest.java
[beam] branch master updated (3918b5f -> 5b1700d)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3918b5f Merge pull request #7688 from robertwb/timestamp-div-fixes new 73f6cb1 [BEAM-6138] Set the PTransform name on Java SDK User counters new 97773b8 spotless new 7c61591 change a comment new e11ee82 added one more test new 8d3d29e spotless new 390fad2 addressed comments new b60a7cc spotless new 5b1700d Merge pull request #7624: [BEAM-6138] Set the PTransform name on Java SDK User counters The 19886 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../fn-execution/src/main/proto/beam_fn_api.proto | 2 + .../core/construction/metrics/MetricKey.java | 4 +- .../runners/core/metrics/MetricsContainerImpl.java | 18 +++- .../core/metrics/MetricsContainerStepMap.java | 40 ++-- .../core/metrics/MonitoringInfoMetricName.java | 1 - .../core/metrics/SimpleMonitoringInfoBuilder.java | 6 ++ .../core/metrics/MetricsContainerImplTest.java | 2 + .../core/metrics/MetricsContainerStepMapTest.java | 50 ++ .../core/metrics/MonitoringInfoTestUtil.java | 44 + .../fnexecution/control/RemoteExecutionTest.java | 44 +++-- sdks/java/harness/build.gradle | 1 + .../beam/fn/harness/BeamFnDataWriteRunner.java | 1 + .../beam/fn/harness/BoundedSourceRunner.java | 2 +- .../org/apache/beam/fn/harness/CombineRunners.java | 3 +- .../fn/harness/DoFnPTransformRunnerFactory.java| 2 + .../org/apache/beam/fn/harness/FlattenRunner.java | 4 +- .../org/apache/beam/fn/harness/MapFnRunners.java | 1 + .../fn/harness/control/ProcessBundleHandler.java | 28 +++--- .../harness/data/ElementCountFnDataReceiver.java | 22 - .../harness/data/PCollectionConsumerRegistry.java | 32 -- .../harness/data/PTransformFunctionRegistry.java | 24 +++-- .../beam/fn/harness/AssignWindowsRunnerTest.java | 7 +- .../beam/fn/harness/BeamFnDataReadRunnerTest.java | 18 +++- .../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 11 ++- .../beam/fn/harness/BoundedSourceRunnerTest.java | 15 ++- .../apache/beam/fn/harness/CombineRunnersTest.java | 45 ++--- .../apache/beam/fn/harness/FlattenRunnerTest.java | 11 ++- .../beam/fn/harness/FnApiDoFnRunnerTest.java | 110 + .../apache/beam/fn/harness/MapFnRunnersTest.java | 39 +--- .../data/ElementCountFnDataReceiverTest.java | 75 +- .../data/PCollectionConsumerRegistryTest.java | 60 +-- .../data/PTransformFunctionRegistryTest.java | 39 +++- 32 files changed, 590 insertions(+), 171 deletions(-) create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MonitoringInfoTestUtil.java
[beam] branch release-2.10.0 updated (d69cd66 -> 5a10bf0)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch release-2.10.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from d69cd66 Merge pull request #7698: [BEAM-6545] Cherrypick #7667 to release-2.10.0: Remove useless allowance of null constructor in ByteArrayShufflePosition new b396e84 [BEAM-6566] Add plugin to check IWYU dependencies, inactivated new ad4d20f [BEAM-6566] Activate IWYU for Beam SQL and fix errors new 5a10bf0 Merge pull request #7707: [BEAM-6558] Cherrypick IWYU fixes The 19506 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: build.gradle | 1 + .../org/apache/beam/gradle/BeamModulePlugin.groovy | 23 ++ sdks/java/extensions/sql/build.gradle | 10 ++ sdks/java/extensions/sql/shell/build.gradle| 5 - 4 files changed, 38 insertions(+), 1 deletion(-)
[beam] branch release-2.10.0 updated: [BEAM-6574] Inline description of CSV formats
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch release-2.10.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.10.0 by this push:
new b73e12d [BEAM-6574] Inline description of CSV formats
new e682cae Merge pull request #7713: [BEAM-6574] Cherrypick #7697 to
release-2.10.0, fixes website precommit
b73e12d is described below
commit b73e12d36271d263535e8b3723c6e5197b7c7049
Author: Kenneth Knowles
AuthorDate: Thu Jan 31 16:49:19 2019 -0800
[BEAM-6574] Inline description of CSV formats
---
.../src/documentation/dsls/sql/create-external-table.md | 15 +--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/website/src/documentation/dsls/sql/create-external-table.md
b/website/src/documentation/dsls/sql/create-external-table.md
index a6f6e32..f332880 100644
--- a/website/src/documentation/dsls/sql/create-external-table.md
+++ b/website/src/documentation/dsls/sql/create-external-table.md
@@ -323,8 +323,19 @@ TBLPROPERTIES '{"format: "Excel"}'
* `LOCATION`: The path to the file for Read Mode. The prefix for Write Mode.
* `TBLPROPERTIES`:
-* `format`: Optional. Allows you to specify the
-
[CSVFormat](https://commons.apache.org/proper/commons-csv/archives/1.5/apidocs/org/apache/commons/csv/CSVFormat.Predefined.html).
+* `format`: Optional. Allows you to specify the CSV Format, which
controls
+the field delimeter, quote character, record separator, and other
properties.
+See the following table:
+
+
+| Value for `format` | Field delimiter | Quote | Record separator | Ignore
empty lines? | Allow missing column names? |
+||-|---|--|-|-|
+| `default` | `,` | `"` | `\r\n` | Yes
| No |
+| `rfc4180` | `,` | `"` | `\r\n` | No
| No |
+| `excel`| `,` | `"` | `\r\n` | No
| Yes |
+| `tdf` | `\t`| `"` | `\r\n` | Yes
| No |
+| `mysql`| `\t`| none | `\n` | No
| No |
+{:.table-bordered}
### Read Mode
[beam] branch master updated (09c556f -> 271d8d0)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 09c556f [BEAM-5959] Adding the gcp_kms_key option to Python SDK. (#7696) new 98cfe65 [BEAM-6566] Add plugin to check IWYU dependencies, inactivated new 297bc1d [BEAM-6566] Activate IWYU for Beam SQL and fix errors new 271d8d0 Merge pull request #7700: [BEAM-6558] Add IWYU plugin, activate for Beam SQL, fix errors The 19876 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: build.gradle | 1 + .../org/apache/beam/gradle/BeamModulePlugin.groovy | 23 ++ sdks/java/extensions/sql/build.gradle | 10 ++ sdks/java/extensions/sql/shell/build.gradle| 5 - 4 files changed, 38 insertions(+), 1 deletion(-)
[beam] branch release-2.10.0 updated: [BEAM-6545] Remove useless allowance of null constructor in ByteArrayShufflePosition
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch release-2.10.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.10.0 by this push:
new 709fff9 [BEAM-6545] Remove useless allowance of null constructor in
ByteArrayShufflePosition
new d69cd66 Merge pull request #7698: [BEAM-6545] Cherrypick #7667 to
release-2.10.0: Remove useless allowance of null constructor in
ByteArrayShufflePosition
709fff9 is described below
commit 709fff940c00082ee3c934ee8dfea75cf420edbb
Author: Kenneth Knowles
AuthorDate: Tue Jan 29 20:51:58 2019 -0800
[BEAM-6545] Remove useless allowance of null constructor in
ByteArrayShufflePosition
---
.../beam/runners/dataflow/worker/GroupingShuffleReader.java | 8 ++--
.../beam/runners/dataflow/worker/PartitioningShuffleReader.java | 8 ++--
.../beam/runners/dataflow/worker/UngroupedShuffleReader.java | 8 ++--
.../worker/util/common/worker/ByteArrayShufflePosition.java | 4 ++--
4 files changed, 20 insertions(+), 8 deletions(-)
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReader.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReader.java
index e840bfb..d2e5bc4 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReader.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReader.java
@@ -235,8 +235,12 @@ public class GroupingShuffleReader extends
NativeReader extends
NativeReader shuffleReader, ShuffleEntryReader
entryReader) {
this.iterator =
entryReader.read(
-
ByteArrayShufflePosition.fromBase64(shuffleReader.startShufflePosition),
-
ByteArrayShufflePosition.fromBase64(shuffleReader.stopShufflePosition));
+ shuffleReader.startShufflePosition == null
+ ? null
+ :
ByteArrayShufflePosition.fromBase64(shuffleReader.startShufflePosition),
+ shuffleReader.stopShufflePosition == null
+ ? null
+ :
ByteArrayShufflePosition.fromBase64(shuffleReader.stopShufflePosition));
this.shuffleReader = shuffleReader;
this.entryReader = entryReader;
}
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReader.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReader.java
index d43c1cd..77d5ae1 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReader.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/UngroupedShuffleReader.java
@@ -88,8 +88,12 @@ public class UngroupedShuffleReader extends
NativeReader {
UngroupedShuffleReader shuffleReader, ShuffleEntryReader
entryReader) {
this.iterator =
entryReader.read(
-
ByteArrayShufflePosition.fromBase64(shuffleReader.startShufflePosition),
-
ByteArrayShufflePosition.fromBase64(shuffleReader.stopShufflePosition));
+ shuffleReader.startShufflePosition == null
+ ? null
+ :
ByteArrayShufflePosition.fromBase64(shuffleReader.startShufflePosition),
+ shuffleReader.stopShufflePosition == null
+ ? null
+ :
ByteArrayShufflePosition.fromBase64(shuffleReader.stopShufflePosition));
this.shuffleReader = shuffleReader;
this.entryReader = entryReader;
}
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ByteArrayShufflePosition.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ByteArrayShufflePosition.java
index 6cd4122..718a3e9 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ByteArrayShufflePosition.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ByteArrayShufflePosition.java
@@ -38,11 +38,11 @@ public class ByteArrayShufflePosition implements
Comparable, Sh
this.position = position;
}
- public static ByteArrayShufflePosition fromBase64(@Nullable String position)
{
+ public static ByteArrayShufflePosition fromBase64(String position) {
return ByteArrayShufflePosition.of(decodeBase64(position));
}
- public static ByteArrayShufflePosition of(@Nullable byte[] position) {
+ public static
[beam] branch master updated: align spotless plugin version across build
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 98f62f5 align spotless plugin version across build
new 42e8f62 Merge pull request #7653: align spotless plugin version
across build
98f62f5 is described below
commit 98f62f5cafb5a9c866c0e0de0e8c101818b83b8b
Author: Michael Luckey <25622840+adude3...@users.noreply.github.com>
AuthorDate: Mon Jan 28 19:19:58 2019 +0100
align spotless plugin version across build
---
build.gradle | 2 +-
buildSrc/build.gradle | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/build.gradle b/build.gradle
index 923a018..ca31165 100644
--- a/build.gradle
+++ b/build.gradle
@@ -54,7 +54,7 @@ buildscript {
classpath "io.spring.gradle:propdeps-plugin:0.0.9.RELEASE"
// Enable provided and optional configurations
classpath "gradle.plugin.org.nosphere.apache:creadur-rat-gradle:0.3.1"
// Enable Apache license enforcement
classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:0.11.0"
// Enable Avro code generation
-classpath "com.diffplug.spotless:spotless-plugin-gradle:3.16.0"
// Enable a code formatting plugin
+classpath "com.diffplug.spotless:spotless-plugin-gradle:3.17.0"
// Enable a code formatting plugin
classpath "gradle.plugin.com.github.blindpirate:gogradle:0.10"
// Enable Go code compilation
classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.20.1"
// Enable building Docker containers
classpath "gradle.plugin.com.dorongold.plugins:task-tree:1.3.1"
// Adds a 'taskTree' task to print task dependency tree
diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle
index 7c34f4c..90731b8 100644
--- a/buildSrc/build.gradle
+++ b/buildSrc/build.gradle
@@ -18,7 +18,7 @@
// Define the set of repositories and dependencies required to
// fetch and enable plugins.
-buildscript { dependencies { classpath
"com.diffplug.spotless:spotless-plugin-gradle:3.15.0" } }
+buildscript { dependencies { classpath
"com.diffplug.spotless:spotless-plugin-gradle:3.17.0" } }
// Plugins for configuring _this build_ of the module
plugins { id 'groovy' }
[beam] branch master updated: [BEAM-6431] Move ExecutionStateSampler and ExecutionStateTracker into runners-core-java
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new ca2a73f [BEAM-6431] Move ExecutionStateSampler and ExecutionStateTracker into runners-core-java new 52e7328 Merge pull request #7634: [BEAM-6431] Move state sampler files to runners-core so they can be used in the Java SDK as well ca2a73f is described below commit ca2a73fd99d713f96e0344fcc56d56e76168d42e Author: Alex Amato AuthorDate: Fri Jan 25 18:41:45 2019 -0800 [BEAM-6431] Move ExecutionStateSampler and ExecutionStateTracker into runners-core-java --- .../core/metrics}/ExecutionStateSampler.java | 9 + .../core/metrics}/ExecutionStateTracker.java | 23 ++ .../core/metrics}/ExecutionStateSamplerTest.java | 6 +++--- .../dataflow/worker/BatchDataflowWorker.java | 2 +- .../dataflow/worker/BatchModeExecutionContext.java | 2 +- .../worker/ChunkingShuffleBatchReader.java | 4 ++-- .../dataflow/worker/ContextActivationObserver.java | 2 +- .../worker/DataflowElementExecutionTracker.java| 2 +- .../dataflow/worker/DataflowExecutionContext.java | 4 ++-- .../worker/DataflowExecutionStateRegistry.java | 2 +- .../dataflow/worker/DataflowMapTaskExecutor.java | 2 +- .../dataflow/worker/DataflowMetricsContainer.java | 2 +- .../dataflow/worker/DataflowOperationContext.java | 4 ++-- .../dataflow/worker/GroupingShuffleReader.java | 4 ++-- .../dataflow/worker/IntrinsicMapTaskExecutor.java | 2 +- ...nmentContextActivationObserverRegistration.java | 2 +- .../beam/runners/dataflow/worker/ShuffleSink.java | 4 ++-- .../dataflow/worker/StreamingDataflowWorker.java | 4 ++-- .../worker/StreamingModeExecutionContext.java | 4 ++-- .../dataflow/worker/WorkItemStatusClient.java | 4 ++-- .../WorkerCustomSourceOperationExecutor.java | 4 ++-- .../worker/fn/control/BeamFnMapTaskExecutor.java | 2 +- .../logging/DataflowWorkerLoggingHandler.java | 4 ++-- .../worker/util/common/worker/MapTaskExecutor.java | 1 + .../util/common/worker/ShuffleReadCounter.java | 1 + .../worker/BatchModeExecutionContextTest.java | 6 +++--- .../ContextActivationObserverRegistryTest.java | 2 +- .../worker/DataflowExecutionContextTest.java | 2 +- .../worker/DataflowExecutionStateTrackerTest.java | 4 ++-- .../worker/DataflowOperationContextTest.java | 10 +- .../worker/DataflowSideInputReadCounterTest.java | 2 +- .../dataflow/worker/GroupingShuffleReaderTest.java | 4 ++-- .../worker/IntrinsicMapTaskExecutorTest.java | 4 ++-- .../runners/dataflow/worker/SimpleParDoFnTest.java | 2 +- .../worker/StreamingModeExecutionContextTest.java | 6 +++--- .../dataflow/worker/TestOperationContext.java | 4 ++-- .../dataflow/worker/WorkItemStatusClientTest.java | 4 ++-- .../dataflow/worker/WorkerCustomSourcesTest.java | 2 +- .../fn/control/BeamFnMapTaskExecutorTest.java | 2 +- .../logging/DataflowWorkerLoggingHandlerTest.java | 2 +- .../worker/GroupingShuffleEntryIteratorTest.java | 2 ++ .../util/common/worker/MapTaskExecutorTest.java| 2 ++ 42 files changed, 91 insertions(+), 69 deletions(-) diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ExecutionStateSampler.java b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateSampler.java similarity index 96% rename from runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ExecutionStateSampler.java rename to runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateSampler.java index d2b437c..c3fb816 100644 --- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/ExecutionStateSampler.java +++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/ExecutionStateSampler.java @@ -15,7 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package org.apache.beam.runners.dataflow.worker.util.common.worker; +package org.apache.beam.runners.core.metrics; import static org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkNotNull; @@ -28,6 +28,7 @@ import java.util.concurrent.Executors; import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; +import javax.annotation.Nullable; import org.apache.beam.vendor.guava.v20_0.com.google.common.annotations.VisibleForTesting; import org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.ThreadFactoryBuilder; import
[beam] branch master updated: add base plugin to py3 container build
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 8430365 add base plugin to py3 container build new daaef01 Merge pull request #7623: apply base plugin to py3 container build 8430365 is described below commit 8430365005c1d29ef0a083c842d365d860e7d824 Author: Michael Luckey <25622840+adude3...@users.noreply.github.com> AuthorDate: Fri Jan 25 13:30:37 2019 +0100 add base plugin to py3 container build --- sdks/python/container/py3/build.gradle | 1 + 1 file changed, 1 insertion(+) diff --git a/sdks/python/container/py3/build.gradle b/sdks/python/container/py3/build.gradle index 9554108..57e4760 100644 --- a/sdks/python/container/py3/build.gradle +++ b/sdks/python/container/py3/build.gradle @@ -16,6 +16,7 @@ * limitations under the License. */ +apply plugin: 'base' apply plugin: org.apache.beam.gradle.BeamModulePlugin applyDockerNature()
[beam] branch master updated (32672b3 -> dcda09a)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 32672b3 Merge pull request #7633: [BEAM-4620] Updates UnboundedReadFromBoundedSource.split() to at least create one split new 16b466c [BEAM-6408] Change test max worker count new a98a717 [BEAM-6408] Increase load test jobs timeout new dcda09a Merge pull request #7485: [BEAM-6408] Fix load test job The 19729 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .test-infra/jenkins/job_LoadTests_Java.groovy | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (941c4cd -> 3f58252)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 941c4cd Merge pull request #7626: align shadow plugin version across build new 8e5dc15 Update org.ajoberstar.grgit dependency to latest (3.0.0) new e8991eb Ensure the website can build without a git repository new 3f58252 Merge pull request #7312: [BEAM-6228] Ensure the website can build without a git repository The 19700 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: build.gradle | 2 +- website/build.gradle | 18 -- 2 files changed, 17 insertions(+), 3 deletions(-)
[beam] branch master updated: align shadow plugin version across build
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 998966d align shadow plugin version across build
new 941c4cd Merge pull request #7626: align shadow plugin version across
build
998966d is described below
commit 998966d264f5314842f9711d1c17ba62b6bd60f3
Author: Michael Luckey <25622840+adude3...@users.noreply.github.com>
AuthorDate: Fri Jan 25 18:52:27 2019 +0100
align shadow plugin version across build
---
build.gradle | 2 +-
buildSrc/build.gradle | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/build.gradle b/build.gradle
index bc9975e..2df3768 100644
--- a/build.gradle
+++ b/build.gradle
@@ -57,8 +57,8 @@ buildscript {
classpath "com.diffplug.spotless:spotless-plugin-gradle:3.16.0"
// Enable a code formatting plugin
classpath "gradle.plugin.com.github.blindpirate:gogradle:0.10"
// Enable Go code compilation
classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.20.1"
// Enable building Docker containers
-classpath "com.github.jengelman.gradle.plugins:shadow:4.0.4"
// Enable shading Java dependencies
classpath "gradle.plugin.com.dorongold.plugins:task-tree:1.3.1"
// Adds a 'taskTree' task to print task dependency tree
+classpath "com.github.jengelman.gradle.plugins:shadow:4.0.3"
// Enable shading Java dependencies
classpath "ca.coglinc:javacc-gradle-plugin:2.4.0"
// Enable the JavaCC parser generator
classpath
"gradle.plugin.io.pry.gradle.offline_dependencies:gradle-offline-dependencies-plugin:0.3"
// Enable creating an offline repository
classpath "net.ltgt.gradle:gradle-errorprone-plugin:0.0.13"
// Enable errorprone Java static analysis
diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle
index 11c4b84..7c34f4c 100644
--- a/buildSrc/build.gradle
+++ b/buildSrc/build.gradle
@@ -32,7 +32,7 @@ repositories { jcenter() }
dependencies {
compile gradleApi()
compile localGroovy()
- compile 'com.github.jengelman.gradle.plugins:shadow:2.0.4'
+ compile 'com.github.jengelman.gradle.plugins:shadow:4.0.3'
}
// Because buildSrc is built and tested automatically _before_ gradle
[beam] branch master updated: Replace visteg plugin with gradle-task-tree to visualize task dependency tree
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new ef48df8 Replace visteg plugin with gradle-task-tree to visualize task
dependency tree
new 40148f5 Merge pull request #7615: Replace visteg plugin with
gradle-task-tree to visualize task dependency tree
ef48df8 is described below
commit ef48df8117052b76dea055daf1c6e22dd1c31022
Author: Scott Wegner
AuthorDate: Thu Jan 24 09:46:15 2019 -0800
Replace visteg plugin with gradle-task-tree to visualize task dependency
tree
Visualizing the Gradle task dependency tree is a useful tool for
debugging build issues. We previously used the visteg plugin, although
the project seems to be abandoned and not compatible with Gradle 5.0
This change switches over to using the gradle-task-tree plugin, which
gives similar functionality (although only prints the tree to the
commandline rather than a .dot file).
Usage:
./gradlew :myProject:myTask :myProject:taskTree
See: https://github.com/dorongold/gradle-task-tree
---
build.gradle | 9 +
.../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 6 ++
2 files changed, 7 insertions(+), 8 deletions(-)
diff --git a/build.gradle b/build.gradle
index cc5db3a..bc9975e 100644
--- a/build.gradle
+++ b/build.gradle
@@ -57,8 +57,8 @@ buildscript {
classpath "com.diffplug.spotless:spotless-plugin-gradle:3.16.0"
// Enable a code formatting plugin
classpath "gradle.plugin.com.github.blindpirate:gogradle:0.10"
// Enable Go code compilation
classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.20.1"
// Enable building Docker containers
-classpath "cz.malohlava:visteg:1.0.3"
// Enable generating Gradle task dependencies as
".dot" files
classpath "com.github.jengelman.gradle.plugins:shadow:4.0.4"
// Enable shading Java dependencies
+classpath "gradle.plugin.com.dorongold.plugins:task-tree:1.3.1"
// Adds a 'taskTree' task to print task dependency tree
classpath "ca.coglinc:javacc-gradle-plugin:2.4.0"
// Enable the JavaCC parser generator
classpath
"gradle.plugin.io.pry.gradle.offline_dependencies:gradle-offline-dependencies-plugin:0.3"
// Enable creating an offline repository
classpath "net.ltgt.gradle:gradle-errorprone-plugin:0.0.13"
// Enable errorprone Java static analysis
@@ -83,13 +83,6 @@ if (!gradle.startParameter.isOffline()) {
apply plugin: "com.gradle.build-scan"
}
-// Apply a task dependency visualization plugin which creates a ".dot" file
containing the
-// task dependencies for the current build. This command can help create a
visual representation:
-// dot -Tsvg build/reports/visteg.dot > build_dependencies.svg
-//
-// See https://github.com/mmalohlava/gradle-visteg for further details.
-apply plugin: "cz.malohlava.visteg"
-
// This plugin provides a task to determine which dependencies have updates.
// Additionally, the plugin checks for updates to Gradle itself.
//
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 3227179..d5a8e82 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -310,6 +310,12 @@ class BeamModulePlugin implements Plugin {
// when attempting to resolve dependency issues.
project.apply plugin: "project-report"
+// Adds a taskTree task that prints task dependency tree report to the
console.
+// Useful for investigating build issues.
+// See: https://github.com/dorongold/gradle-task-tree
+project.apply plugin: "com.dorongold.task-tree"
+project.taskTree { noRepeat = true }
+
/**
***/
// Define and export a map dependencies shared across multiple
sub-projects.
//
[beam] branch master updated: Add a bit more details to the CONTRIBUTING stub
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new bcf9506 Add a bit more details to the CONTRIBUTING stub new cfe94db Merge pull request #7628: Add a bit more details to the CONTRIBUTING stub bcf9506 is described below commit bcf95063f113dab49ad33f843b429cd470d6e362 Author: Scott Wegner AuthorDate: Fri Jan 25 10:59:13 2019 -0800 Add a bit more details to the CONTRIBUTING stub --- CONTRIBUTING.md | 11 --- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 44b0da5..2784be4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -19,6 +19,11 @@ # How to Contribute -The Apache Beam community welcomes contributions from anyone with a passion for -data processing! Please see our [contribution guide](https://beam.apache.org/contribute/contribution-guide/) -for details. +The Apache Beam community welcomes contributions from anyone! +Please see our [contribution guide](https://beam.apache.org/contribute/contribution-guide/) +for details, such as: + +* Sharing your intent with the community +* Development setup and testing your changes +* Submitting a pull request and finding a reviewer +
[beam] branch master updated: Add additional debug output for metrics prober test failures
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8317d61 Add additional debug output for metrics prober test failures
new b80fc41 Merge pull request #7625: [BEAM-6381] Add additional debug
output for metrics prober test failures
8317d61 is described below
commit 8317d61b6b0f61e57a33e3135baab6b712eea33f
Author: Scott Wegner
AuthorDate: Fri Jan 25 09:43:43 2019 -0800
Add additional debug output for metrics prober test failures
---
.test-infra/metrics/src/test/groovy/ProberTests.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.test-infra/metrics/src/test/groovy/ProberTests.groovy
b/.test-infra/metrics/src/test/groovy/ProberTests.groovy
index 4b65dbe..a0c8481 100644
--- a/.test-infra/metrics/src/test/groovy/ProberTests.groovy
+++ b/.test-infra/metrics/src/test/groovy/ProberTests.groovy
@@ -45,7 +45,7 @@ class ProberTests {
def alerts = new JsonSlurper().parseText(alertsJson)
assert alerts.size > 0
alerts.each { alert ->
- assert alert.state == 'ok' : "Input data is stale!
${grafanaEndpoint}/d/data-freshness"
+ assert alert.state == 'ok' : "Input data is stale! ${alert}\n See:
${grafanaEndpoint}/d/data-freshness"
}
}
}
[beam] branch master updated: [Dependency Update] upgrade to shadow plugin 4.0.4
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new db6a490 [Dependency Update] upgrade to shadow plugin 4.0.4
new 00be31e Merge pull request #7622: [Dependency Update] upgrade to
shadow plugin 4.0.4
db6a490 is described below
commit db6a4903ea44ab3684de4c84064781375ad03e28
Author: Michael Luckey <25622840+adude3...@users.noreply.github.com>
AuthorDate: Fri Jan 25 02:11:56 2019 +0100
[Dependency Update] upgrade to shadow plugin 4.0.4
---
build.gradle | 2 +-
runners/flink/job-server/flink_job_server.gradle | 7 ---
runners/reference/job-server/build.gradle| 8
3 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/build.gradle b/build.gradle
index bb9de48..98856c9 100644
--- a/build.gradle
+++ b/build.gradle
@@ -58,7 +58,7 @@ buildscript {
classpath "gradle.plugin.com.github.blindpirate:gogradle:0.10"
// Enable Go code compilation
classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.20.1"
// Enable building Docker containers
classpath "cz.malohlava:visteg:1.0.3"
// Enable generating Gradle task dependencies as
".dot" files
-classpath "com.github.jengelman.gradle.plugins:shadow:2.0.4"
// Enable shading Java dependencies
+classpath "com.github.jengelman.gradle.plugins:shadow:4.0.4"
// Enable shading Java dependencies
classpath "ca.coglinc:javacc-gradle-plugin:2.4.0"
// Enable the JavaCC parser generator
classpath
"gradle.plugin.io.pry.gradle.offline_dependencies:gradle-offline-dependencies-plugin:0.3"
// Enable creating an offline repository
classpath "net.ltgt.gradle:gradle-errorprone-plugin:0.0.13"
// Enable errorprone Java static analysis
diff --git a/runners/flink/job-server/flink_job_server.gradle
b/runners/flink/job-server/flink_job_server.gradle
index 7620bb6..dd114bc 100644
--- a/runners/flink/job-server/flink_job_server.gradle
+++ b/runners/flink/job-server/flink_job_server.gradle
@@ -25,6 +25,10 @@ import org.apache.beam.gradle.BeamModulePlugin
*/
apply plugin: org.apache.beam.gradle.BeamModulePlugin
+apply plugin: "application"
+// we need to set mainClassName before applying shadow plugin
+mainClassName = "org.apache.beam.runners.flink.FlinkJobServerDriver"
+
applyJavaNature(
validateShadowJar: false,
exportJavadoc: false,
@@ -32,7 +36,6 @@ applyJavaNature(
append "reference.conf"
},
)
-apply plugin: "application"
// Resolve the Flink project name (and version) the job-server is based on
def flinkRunnerProject = ":${project.name.replace("-job-server", "")}"
@@ -62,8 +65,6 @@ sourceSets {
}
}
-mainClassName = "org.apache.beam.runners.flink.FlinkJobServerDriver"
-
configurations {
validatesPortableRunner
}
diff --git a/runners/reference/job-server/build.gradle
b/runners/reference/job-server/build.gradle
index 6e713bc..e476d6a 100644
--- a/runners/reference/job-server/build.gradle
+++ b/runners/reference/job-server/build.gradle
@@ -16,6 +16,10 @@
* limitations under the License.
*/
apply plugin: org.apache.beam.gradle.BeamModulePlugin
+apply plugin: "application"
+// we need to set mainClassName before applying shadow plugin
+mainClassName =
"org.apache.beam.runners.direct.portable.job.ReferenceRunnerJobServer"
+
applyJavaNature(
exportJavadoc: false,
validateShadowJar: false,
@@ -25,10 +29,6 @@ applyJavaNature(
description = "Apache Beam :: Runners :: Reference :: Job Server"
-apply plugin: "application"
-
-mainClassName =
"org.apache.beam.runners.direct.portable.job.ReferenceRunnerJobServer"
-
dependencies {
compile project(path: ":beam-runners-direct-java", configuration: "shadow")
compile project(path: ":beam-runners-java-fn-execution", configuration:
"shadow")
[beam] branch master updated: Refactor, Remove references to Dataflow classes in base State Sampling classes.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 38adb38 Refactor, Remove references to Dataflow classes in base State
Sampling classes.
new e811212 Merge pull request #7507: [BEAM-6431] Refactor, Remove
references to Dataflow classes in base State Sampling
38adb38 is described below
commit 38adb38a6e1908756208bf01d6a8339529c86469
Author: Alex Amato
AuthorDate: Mon Jan 14 17:44:16 2019 -0800
Refactor, Remove references to Dataflow classes in base State Sampling
classes.
---
.../dataflow/worker/BatchModeExecutionContext.java | 8 +-
.../dataflow/worker/DataflowExecutionContext.java | 36 +-
...tateKey.java => DataflowExecutionStateKey.java} | 11 +-
...ry.java => DataflowExecutionStateRegistry.java} | 14 +--
.../dataflow/worker/DataflowOperationContext.java | 18 ++-
.../worker/StreamingModeExecutionContext.java | 7 +-
.../logging/DataflowWorkerLoggingHandler.java | 11 +-
.../common/worker/ElementExecutionTracker.java | 3 +-
.../util/common/worker/ExecutionStateTracker.java | 42 ++-
.../util/common/worker/ShuffleReadCounter.java | 6 +-
.../worker/BatchModeExecutionContextTest.java | 4 +-
...java => DataflowExecutionStateTrackerTest.java} | 128 +
.../worker/DataflowOperationContextTest.java | 2 +-
.../worker/DataflowSideInputReadCounterTest.java | 8 +-
.../dataflow/worker/GroupingShuffleReaderTest.java | 25 ++--
.../worker/IntrinsicMapTaskExecutorTest.java | 6 +-
.../worker/StreamingModeExecutionContextTest.java | 4 +-
.../dataflow/worker/TestOperationContext.java | 25 +++-
.../dataflow/worker/WorkItemStatusClientTest.java | 1 -
.../logging/DataflowWorkerLoggingHandlerTest.java | 12 +-
.../common/worker/ExecutionStateSamplerTest.java | 93 ++-
.../worker/GroupingShuffleEntryIteratorTest.java | 25 ++--
.../util/common/worker/MapTaskExecutorTest.java| 6 +-
23 files changed, 175 insertions(+), 320 deletions(-)
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContext.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContext.java
index c879352..6d42a10 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContext.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContext.java
@@ -74,7 +74,7 @@ public class BatchModeExecutionContext
ReaderFactory readerFactory,
PipelineOptions options,
DataflowExecutionStateTracker executionStateTracker,
- ExecutionStateRegistry executionStateRegistry) {
+ DataflowExecutionStateRegistry executionStateRegistry) {
super(
counterFactory,
createMetricsContainerRegistry(),
@@ -192,9 +192,11 @@ public class BatchModeExecutionContext
}
}
- /** {@link ExecutionStateRegistry} that creates {@link
BatchModeExecutionState} instances. */
+ /**
+ * {@link DataflowExecutionStateRegistry} that creates {@link
BatchModeExecutionState} instances.
+ */
@VisibleForTesting
- public static class BatchModeExecutionStateRegistry extends
ExecutionStateRegistry {
+ public static class BatchModeExecutionStateRegistry extends
DataflowExecutionStateRegistry {
@Override
protected DataflowOperationContext.DataflowExecutionState createState(
diff --git
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionContext.java
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionContext.java
index fa73b15..621cdb5 100644
---
a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionContext.java
+++
b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionContext.java
@@ -32,8 +32,10 @@ import org.apache.beam.runners.core.SideInputReader;
import org.apache.beam.runners.core.StepContext;
import org.apache.beam.runners.core.TimerInternals.TimerData;
import
org.apache.beam.runners.dataflow.worker.DataflowExecutionContext.DataflowStepContext;
+import
org.apache.beam.runners.dataflow.worker.DataflowOperationContext.DataflowExecutionState;
import org.apache.beam.runners.dataflow.worker.counters.CounterFactory;
import org.apache.beam.runners.dataflow.worker.counters.NameContext;
+import
org.apache.beam.runners.dataflow.worker.util.common.worker.ElementExecutionTracker
[beam] branch master updated: UnboundedReadFromBoundedSource should invoke split for small sources
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f22f443 UnboundedReadFromBoundedSource should invoke split for small
sources
new 0aae7e5 Merge pull request #7555: [BEAM-4620]
UnboundedReadFromBoundedSource invokes split for small bounded sources
f22f443 is described below
commit f22f443c51ed40a923aa81a73a48492b578c8583
Author: Chamikara Jayalath
AuthorDate: Thu Jan 17 18:19:40 2019 -0800
UnboundedReadFromBoundedSource should invoke split for small sources
---
.../construction/UnboundedReadFromBoundedSource.java | 20 +++-
.../UnboundedReadFromBoundedSourceTest.java | 11 +++
2 files changed, 26 insertions(+), 5 deletions(-)
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
index e42487c..47bceeb 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
@@ -74,6 +74,9 @@ public class UnboundedReadFromBoundedSource extends
PTransform source;
/**
@@ -124,13 +127,20 @@ public class UnboundedReadFromBoundedSource extends
PTransform> split(
int desiredNumSplits, PipelineOptions options) throws Exception {
try {
-long desiredBundleSize = boundedSource.getEstimatedSizeBytes(options)
/ desiredNumSplits;
-if (desiredBundleSize <= 0) {
+long estimatedSize = boundedSource.getEstimatedSizeBytes(options);
+if (estimatedSize <= 0) {
+ // Source is unable to provide a valid estimated size. So using
default size.
LOG.warn(
- "BoundedSource {} cannot estimate its size, skips the initial
splits.",
- boundedSource);
- return ImmutableList.of(this);
+ "Cannot determine a valid estimated size for BoundedSource {}.
Using default "
+ + "size of {} bytes",
+ boundedSource,
+ DEFAULT_ESTIMATED_SIZE);
+ estimatedSize = DEFAULT_ESTIMATED_SIZE;
}
+
+// Each split should at least be of size 1 byte.
+long desiredBundleSize = Math.max(estimatedSize / desiredNumSplits, 1);
+
List> splits =
boundedSource.split(desiredBundleSize, options);
return splits.stream()
.map(input -> new BoundedToUnboundedSourceAdapter<>(input))
diff --git
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
index 42e036b..3f8671d 100644
---
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
+++
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
@@ -18,6 +18,7 @@
package org.apache.beam.runners.core.construction;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertTrue;
@@ -245,6 +246,16 @@ public class UnboundedReadFromBoundedSourceTest {
}
@Test
+ public void testInvokesSplitWithDefaultNumSplitsTooLarge() throws Exception {
+UnboundedSource unboundedCountingSource =
+new BoundedToUnboundedSourceAdapter(CountingSource.upTo(1));
+PipelineOptions options = PipelineOptionsFactory.create();
+List splits = unboundedCountingSource.split(100, options);
+assertEquals(1, splits.size());
+assertNotEquals(splits.get(0), unboundedCountingSource);
+ }
+
+ @Test
public void testReadFromCheckpointBeforeStart() throws Exception {
thrown.expect(NoSuchElementException.class);
[beam] branch master updated: [Dependency Update] upgrade to gradle-apt-plugin 0.20
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8d5a368 [Dependency Update] upgrade to gradle-apt-plugin 0.20
new 5eaf33c Merge pull request #7613: [Dependency Update] upgrade to
gradle-apt-plugin 0.20
8d5a368 is described below
commit 8d5a368d2fecf1a4db46b8591bc3b6a680eed9b2
Author: Michael Luckey <25622840+adude3...@users.noreply.github.com>
AuthorDate: Thu Jan 24 15:43:49 2019 +0100
[Dependency Update] upgrade to gradle-apt-plugin 0.20
---
build.gradle | 2 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 42 +-
2 files changed, 9 insertions(+), 35 deletions(-)
diff --git a/build.gradle b/build.gradle
index bb9de48..22b0c77 100644
--- a/build.gradle
+++ b/build.gradle
@@ -49,7 +49,7 @@ buildscript {
}
dependencies {
classpath 'net.researchgate:gradle-release:2.6.0'
// Enable gradle-based release management
-classpath "net.ltgt.gradle:gradle-apt-plugin:0.13"
// Enable a Java annotation processor
+classpath "net.ltgt.gradle:gradle-apt-plugin:0.20"
// Enable a Java annotation processor
classpath "com.google.protobuf:protobuf-gradle-plugin:0.8.5"
// Enable proto code generation
classpath "io.spring.gradle:propdeps-plugin:0.0.9.RELEASE"
// Enable provided and optional configurations
classpath "gradle.plugin.org.nosphere.apache:creadur-rat-gradle:0.3.1"
// Enable Apache license enforcement
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 7340f8f..859a896 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -657,6 +657,8 @@ class BeamModulePlugin implements Plugin {
// Configures annotation processing for commonly used annotation
processors
// across all Java projects.
project.apply plugin: "net.ltgt.apt"
+ // let idea apt plugin handle the ide integration
+ project.apply plugin: "net.ltgt.apt-idea"
project.dependencies {
// Note that these plugins specifically use the compileOnly and
testCompileOnly
// configurations because they are never required to be shaded or
become a
@@ -667,13 +669,13 @@ class BeamModulePlugin implements Plugin {
compileOnly auto_value_annotations
testCompileOnly auto_value_annotations
-apt auto_value
-testApt auto_value
+annotationProcessor auto_value
+testAnnotationProcessor auto_value
compileOnly auto_service
testCompileOnly auto_service
-apt auto_service
-testApt auto_service
+annotationProcessor auto_service
+testAnnotationProcessor auto_service
// These dependencies are needed to avoid error-prone warnings on
package-info.java files,
// also to include the annotations to suppress warnings.
@@ -684,8 +686,8 @@ class BeamModulePlugin implements Plugin {
def findbugs_annotations = "com.google.code.findbugs:annotations:3.0.1"
compileOnly findbugs_annotations
testCompileOnly findbugs_annotations
-apt findbugs_annotations
-testApt findbugs_annotations
+annotationProcessor findbugs_annotations
+testAnnotationProcessor findbugs_annotations
}
// Add the optional and provided configurations for dependencies
@@ -1089,34 +1091,6 @@ class BeamModulePlugin implements Plugin {
}
}
}
-
- // These directories for when build actions are delegated to Gradle
- def gradleAptGeneratedMain =
"${project.buildDir}/generated/source/apt/main"
- def gradleAptGeneratedTest =
"${project.buildDir}/generated/source/apt/test"
-
- // These directories for when build actions are executed by Idea
- // IntelliJ does not add these source roots (that it owns!) unless hinted
- def ideaRoot = "${project.projectDir}/out"
- def ideaAptGeneratedMain = "${ideaRoot}/production/classes/generated"
- def ideaAptGeneratedTest = "${ideaRoot}/test/classes/generated_test"
-
- project.idea {
-module {
- sourceDirs += project.file(gradleAptGeneratedMain)
- testSourceDirs += project.file(gradleAptGeneratedTest)
-
-
[beam] branch master updated: [BEAM-6161] Introduce PCollectionConsumerRegistry and add ElementCount counters to the java SDK
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f41b2ef [BEAM-6161] Introduce PCollectionConsumerRegistry and add
ElementCount counters to the java SDK
new fb13cb1 Merge pull request #7272: [BEAM-6161] Introduce
PCollectionConsumerRegistry and add ElementCoun…
f41b2ef is described below
commit f41b2ef319d396479d083046280500ad1cbe0be6
Author: Alex Amato
AuthorDate: Wed Dec 12 17:37:45 2018 -0800
[BEAM-6161] Introduce PCollectionConsumerRegistry and add ElementCount
counters to the java SDK
---
.../fn-execution/src/main/proto/beam_fn_api.proto | 10 +-
.../beam/runners/core/metrics/LabeledMetrics.java | 35 ++
.../runners/core/metrics/MetricsContainerImpl.java | 37 --
.../core/metrics/MonitoringInfoMetricName.java | 124
.../core/metrics/SimpleMonitoringInfoBuilder.java | 33 +-
.../core/metrics/SpecMonitoringInfoValidator.java | 1 +
.../runners/core/metrics/LabeledMetricsTest.java | 76
.../core/metrics/MetricsContainerImplTest.java | 53 +
.../core/metrics/MonitoringInfoMetricNameTest.java | 127 +
.../metrics/SimpleMonitoringInfoBuilderTest.java | 8 +-
.../metrics/SpecMonitoringInfoValidatorTest.java | 2 +-
.../fnexecution/control/RemoteExecutionTest.java | 78 +++--
.../apache/beam/sdk/metrics/DelegatingCounter.java | 64 +++
.../org/apache/beam/sdk/metrics/MetricResult.java | 5 +-
.../java/org/apache/beam/sdk/metrics/Metrics.java | 41 ---
.../beam/fn/harness/BeamFnDataReadRunner.java | 25 ++--
.../beam/fn/harness/BeamFnDataWriteRunner.java | 7 +-
.../beam/fn/harness/BoundedSourceRunner.java | 10 +-
.../org/apache/beam/fn/harness/CombineRunners.java | 21 ++--
.../fn/harness/DoFnPTransformRunnerFactory.java| 19 +--
.../org/apache/beam/fn/harness/FlattenRunner.java | 15 +--
.../org/apache/beam/fn/harness/MapFnRunners.java | 20 ++--
.../beam/fn/harness/PTransformRunnerFactory.java | 8 +-
.../fn/harness/control/ProcessBundleHandler.java | 21 ++--
.../harness/data/ElementCountFnDataReceiver.java | 54 +
.../harness/data/PCollectionConsumerRegistry.java | 103 +
.../beam/fn/harness/AssignWindowsRunnerTest.java | 13 +--
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 13 ++-
.../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 8 +-
.../beam/fn/harness/BoundedSourceRunnerTest.java | 10 +-
.../apache/beam/fn/harness/CombineRunnersTest.java | 31 +++--
.../apache/beam/fn/harness/FlattenRunnerTest.java | 26 ++---
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 106 -
.../apache/beam/fn/harness/MapFnRunnersTest.java | 24 ++--
.../harness/control/ProcessBundleHandlerTest.java | 17 ++-
.../data/ElementCountFnDataReceiverTest.java | 70
.../data/PCollectionConsumerRegistryTest.java | 88 ++
37 files changed, insertions(+), 292 deletions(-)
diff --git a/model/fn-execution/src/main/proto/beam_fn_api.proto
b/model/fn-execution/src/main/proto/beam_fn_api.proto
index 623f23d..378de46 100644
--- a/model/fn-execution/src/main/proto/beam_fn_api.proto
+++ b/model/fn-execution/src/main/proto/beam_fn_api.proto
@@ -322,11 +322,6 @@ message MonitoringInfoSpec {
repeated Annotation annotations = 4;
}
-extend google.protobuf.EnumValueOptions {
- // Enum extension to store the MonitoringInfoSpecs.
- MonitoringInfoSpec monitoring_info_spec = 207174266;
-}
-
// The key name and value string of MonitoringInfo annotations.
message Annotation {
string key = 1;
@@ -348,7 +343,7 @@ message MonitoringInfoSpecs {
ELEMENT_COUNT = 1 [(monitoring_info_spec) = {
urn: "beam:metric:element_count:v1",
type_urn: "beam:metrics:sum_int_64",
- required_labels: [ "PTRANSFORM", "PCOLLECTION" ],
+ required_labels: [ "PCOLLECTION" ],
annotations: [ {
key: "description",
value: "The total elements output to a Pcollection by a PTransform."
@@ -411,6 +406,9 @@ message MonitoringInfoLabelProps {
// specifications, constants, etc.
extend google.protobuf.EnumValueOptions {
MonitoringInfoLabelProps label_props = 127337796; // From: commit 0x7970544.
+
+ // Enum extension to store the MonitoringInfoSpecs.
+ MonitoringInfoSpec monitoring_info_spec = 207174266;
}
message MonitoringInfo {
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/LabeledMetrics.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/LabeledMetrics.java
new file mode 100644
index 000..bf6295b
--- /dev/null
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/cor
[beam] branch master updated: Add missing javadocs for new public methods
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 4fedc75 Add missing javadocs for new public methods
new 98952d2 Merge pull request #7618: [BEAM-6500] Add missing javadocs
for new public methods
4fedc75 is described below
commit 4fedc75194e24d9b89b8d6326143136a25126b91
Author: Scott Wegner
AuthorDate: Thu Jan 24 10:30:53 2019 -0800
Add missing javadocs for new public methods
---
.../java/org/apache/beam/runners/direct/portable/ReferenceRunner.java| 1 +
.../beam/runners/direct/portable/job/ReferenceRunnerJobServer.java | 1 +
2 files changed, 2 insertions(+)
diff --git
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
index 5458a21..07212a3 100644
---
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
+++
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
@@ -161,6 +161,7 @@ public class ReferenceRunner {
return res;
}
+ /** Configures and starts the {@link ExecutorServiceParallelExecutor}. */
public void execute() throws Exception {
ExecutableGraph graph =
PortableGraph.forPipeline(pipeline);
BundleFactory bundleFactory = ImmutableListBundleFactory.create();
diff --git
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServer.java
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServer.java
index 31d84e8..e2eac87 100644
---
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServer.java
+++
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServer.java
@@ -144,6 +144,7 @@ public class ReferenceRunnerJobServer {
return jobServiceConfig;
}
+ /** Command-line options to configure the JobServer. */
public static class ServerConfiguration {
@Option(
name = "-p",
[beam] branch master updated: Fix typo in github sync script
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new e32f90f Fix typo in github sync script new 3272889 Merge pull request #7607: Fix typo in github sync script e32f90f is described below commit e32f90fa225ae65d13e0c734a65913bee03d53b3 Author: Mikhail Gryzykhin AuthorDate: Wed Jan 23 11:07:24 2019 -0800 Fix typo in github sync script --- .test-infra/metrics/sync/github/sync.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.test-infra/metrics/sync/github/sync.py b/.test-infra/metrics/sync/github/sync.py index 1b67d98..930275b 100644 --- a/.test-infra/metrics/sync/github/sync.py +++ b/.test-infra/metrics/sync/github/sync.py @@ -46,7 +46,7 @@ DB_NAME = os.environ['DB_DBNAME'] DB_USER_NAME = os.environ['DB_DBUSERNAME'] DB_PASSWORD = os.environ['DB_DBPWD'] -GH_ACCESS_TOKEN = os.environ['GH_ACCESSTOKEN'] +GH_ACCESS_TOKEN = os.environ['GH_ACCESS_TOKEN'] GH_PRS_TABLE_NAME = 'gh_pull_requests'
[beam] branch master updated: [BEAM-6237] Fix ULR not deleting artifacts after running jobs.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 0e34795 [BEAM-6237] Fix ULR not deleting artifacts after running jobs.
new da66729 Merge pull request #7571: [BEAM-6237] Fix ULR not deleting
artifacts after running jobs.
0e34795 is described below
commit 0e347955685c8d9310ee3ea8efa0f9268425de3d
Author: Daniel Oliveira
AuthorDate: Fri Dec 21 16:37:48 2018 -0800
[BEAM-6237] Fix ULR not deleting artifacts after running jobs.
This change switches the ULR from using
LocalFileSystemArtifactStagerService to using
BeamFileSystemArtifactStagingService which has functionality to remove
artifacts after running a job. With this change ValidatesRunner tests
no longer leave huge amounts of artifacts when run with the ULR.
---
.../runners/direct/portable/ReferenceRunner.java | 39 +-
.../runners/direct/portable/job/PreparingJob.java | 11 ++-
.../portable/job/ReferenceRunnerJobServer.java | 37 +-
.../portable/job/ReferenceRunnerJobService.java| 85 +-
.../job/ReferenceRunnerJobServiceTest.java | 11 ++-
5 files changed, 118 insertions(+), 65 deletions(-)
diff --git
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
index 2c2c9f0..4824bcb 100644
---
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
+++
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
@@ -22,14 +22,12 @@ import static
org.apache.beam.vendor.guava.v20_0.com.google.common.base.Precondi
import static
org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
import static
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Iterables.getOnlyElement;
-import java.io.File;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.stream.Collectors;
-import javax.annotation.Nullable;
import org.apache.beam.model.fnexecution.v1.ProvisionApi.ProvisionInfo;
import org.apache.beam.model.fnexecution.v1.ProvisionApi.Resources;
import org.apache.beam.model.pipeline.v1.RunnerApi;
@@ -56,13 +54,12 @@ import
org.apache.beam.runners.core.construction.graph.ProtoOverrides;
import
org.apache.beam.runners.core.construction.graph.ProtoOverrides.TransformReplacement;
import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
import org.apache.beam.runners.direct.ExecutableGraph;
-import
org.apache.beam.runners.direct.portable.artifact.LocalFileSystemArtifactRetrievalService;
-import
org.apache.beam.runners.direct.portable.artifact.UnsupportedArtifactRetrievalService;
import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.InProcessServerFactory;
import org.apache.beam.runners.fnexecution.ServerFactory;
import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import
org.apache.beam.runners.fnexecution.artifact.BeamFileSystemArtifactRetrievalService;
import org.apache.beam.runners.fnexecution.control.ControlClientPool;
import
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService;
import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
@@ -90,31 +87,42 @@ import
org.apache.beam.vendor.guava.v20_0.com.google.common.collect.Sets;
import org.joda.time.Duration;
import org.joda.time.Instant;
-/** The "ReferenceRunner" engine implementation. */
+/**
+ * The "ReferenceRunner" engine implementation. The ReferenceRunner uses the
portability framework
+ * to execute a Pipeline on a single machine.
+ */
public class ReferenceRunner {
private final RunnerApi.Pipeline pipeline;
private final Struct options;
- @Nullable private final File artifactsDir;
+ private final String artifactRetrievalToken;
private final EnvironmentType environmentType;
private final IdGenerator idGenerator = IdGenerators.incrementingLongs();
+ /** @param environmentType The environment to use for the SDK Harness. */
private ReferenceRunner(
- Pipeline p, Struct options, @Nullable File artifactsDir, EnvironmentType
environmentType) {
+ Pipeline p, Struct options, String artifactRetrievalToken,
EnvironmentType environmentType) {
this.pipeline = executable(p);
this.options = options;
-this.artifactsDir = artifactsDir;
this.environmentType = environmentType;
+this.artifactRetrievalToken = artifactRetrievalToken;
}
+ /**
+ * Creates a &quo
[beam] branch master updated (26173a8 -> 21f4b54)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 26173a8 Merge pull request #7497: [BEAM-6351] Divide separate job for "smoke" load test suites new 47ad785 Revert "[BEAM-2939] SplittableDoFn Java SDK API Changes (#6969)" new 4fa1ac2 Do not revert Backlog and Backlogs new 0c5df62 spotless new 21f4b54 Merge pull request #7600: [BEAM-6354] Revert "[BEAM-2939] SplittableDoFn Java SDK API Changes (#6969)" The 19631 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../fn-execution/src/main/proto/beam_fn_api.proto | 17 ++- .../runners/core/construction/SplittableParDo.java | 2 - .../construction/SplittableParDoNaiveBounded.java | 4 +- .../core/SplittableParDoViaKeyedWorkItems.java | 7 +-- .../runners/core/SplittableParDoProcessFnTest.java | 18 .../direct/portable/ReferenceRunnerTest.java | 3 +- .../java/org/apache/beam/sdk/transforms/DoFn.java | 49 +--- .../reflect/ByteBuddyDoFnInvokerFactory.java | 6 +-- .../beam/sdk/transforms/reflect/DoFnInvoker.java | 2 - .../sdk/transforms/reflect/DoFnSignatures.java | 13 ++ .../splittabledofn/RestrictionTracker.java | 9 +--- .../transforms/splittabledofn/Restrictions.java| 29 .../beam/sdk/transforms/SplittableDoFnTest.java| 5 +-- .../sdk/transforms/reflect/DoFnInvokersTest.java | 20 + .../reflect/DoFnSignaturesSplittableDoFnTest.java | 52 +++--- .../harness/SplittableProcessElementsRunner.java | 4 +- .../beam/sdk/io/hbase/HBaseReadSplittableDoFn.java | 3 +- 17 files changed, 39 insertions(+), 204 deletions(-) delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Restrictions.java
[beam] branch master updated (a843a08 -> 8b4898e)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from a843a08 Merge pull request #7534 Native type hints in Beam type hints
new 150f5c7 Implemented msec counters support in FnApi world.
new 85970a0 run checkstyle, spotless, etc
new 38c6367 Use constants
new 8b4898e Merge pull request #7323: [BEAM-6181] Implemented msec
counters support in FnApi world.
The 19621 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../core/metrics/SimpleMonitoringInfoBuilder.java | 84 +++
.../core/metrics/SpecMonitoringInfoValidator.java | 81 ++
.../metrics/SpecMonitoringInfoValidatorTest.java | 77 ++
.../dataflow/worker/WorkItemStatusClient.java | 31 +++-
.../worker/fn/control/BeamFnMapTaskExecutor.java | 95 +---
...piMonitoringInfoToCounterUpdateTransformer.java | 71 +
...ecMonitoringInfoToCounterUpdateTransformer.java | 144 ++
.../MonitoringInfoToCounterUpdateTransformer.java} | 16 +-
...erMonitoringInfoToCounterUpdateTransformer.java | 133
.../fn/control/BeamFnMapTaskExecutorTest.java | 14 +-
...nitoringInfoToCounterUpdateTransformerTest.java | 90 +++
...nitoringInfoToCounterUpdateTransformerTest.java | 167 +
...nitoringInfoToCounterUpdateTransformerTest.java | 117 +++
13 files changed, 947 insertions(+), 173 deletions(-)
create mode 100644
runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidator.java
create mode 100644
runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SpecMonitoringInfoValidatorTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/FnApiMonitoringInfoToCounterUpdateTransformer.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformer.java
copy
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/{DataflowWorkExecutor.java
=> fn/control/MonitoringInfoToCounterUpdateTransformer.java} (65%)
create mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformer.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/FnApiMonitoringInfoToCounterUpdateTransformerTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/MSecMonitoringInfoToCounterUpdateTransformerTest.java
create mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/UserMonitoringInfoToCounterUpdateTransformerTest.java
[beam] branch master updated (fa3af06 -> 324a1bc)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fa3af06 Merge pull request #7591 from lostluck/singlewindow new abd6873 [BEAM-6491] Ignore flaky test: FileIOTest.testMatchWatchForNewFiles new 729b340 [BEAM-6491] Fixup new 324a1bc Merge pull request #7595: [BEAM-6491] Ignore flaky test: FileIOTest.testMatchWatchForNewFiles The 19611 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java | 2 ++ 1 file changed, 2 insertions(+)
[beam] branch master updated: Add named variant of PTransform#compose()
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 65c8d51 Add named variant of PTransform#compose()
new 1991965 Merge pull request #7437: [BEAM-6386] Add named variant of
PTransform::compose()
65c8d51 is described below
commit 65c8d510d6e867c7da1320405c1f4e5e18d9c90d
Author: Jeff Klukas
AuthorDate: Tue Jan 8 13:16:13 2019 -0500
Add named variant of PTransform#compose()
---
.../main/java/org/apache/beam/sdk/transforms/PTransform.java | 12
.../java/org/apache/beam/sdk/transforms/PTransformTest.java | 8
2 files changed, 20 insertions(+)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/PTransform.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/PTransform.java
index 49c40f7..0c6b4d2 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/PTransform.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/PTransform.java
@@ -319,4 +319,16 @@ public abstract class PTransform
}
};
}
+
+ /** Like {@link #compose(SerializableFunction)}, but with a custom name. */
+ @Experimental
+ public static
+ PTransform compose(String name,
SerializableFunction fn) {
+return new PTransform(name) {
+ @Override
+ public OutputT expand(InputT input) {
+return fn.apply(input);
+ }
+};
+ }
}
diff --git
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/PTransformTest.java
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/PTransformTest.java
index 35d18d9..ae63d5b 100644
---
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/PTransformTest.java
+++
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/PTransformTest.java
@@ -19,6 +19,7 @@ package org.apache.beam.sdk.transforms;
import static org.apache.beam.sdk.values.TypeDescriptors.integers;
import static org.hamcrest.Matchers.empty;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertThat;
import java.io.Serializable;
@@ -54,6 +55,13 @@ public class PTransformTest implements Serializable {
}
@Test
+ public void testNamedCompose() {
+PTransform, PCollection> composed =
+PTransform.compose("MyName", (PCollection numbers) ->
numbers);
+assertEquals("MyName", composed.name);
+ }
+
+ @Test
@Category(NeedsRunner.class)
public void testComposeBasicSerializableFunction() throws Exception {
PCollection output =
[beam] branch release-2.10.0 updated: [BEAM-6352] Revert PR#6467 to fix Watch transform from swegner/revert_pr6467
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch release-2.10.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.10.0 by this push:
new b352df3 [BEAM-6352] Revert PR#6467 to fix Watch transform from
swegner/revert_pr6467
new bfd6893 Merge pull request #7575: [BEAM-6352] Revert PR#6467 to fix
Watch transform
b352df3 is described below
commit b352df3976c6a687213b611a9a47b6e6093a2dc5
Author: Kenn Knowles
AuthorDate: Fri Jan 18 13:33:16 2019 -0800
[BEAM-6352] Revert PR#6467 to fix Watch transform from swegner/revert_pr6467
---
.../runners/apex/translation/ParDoTranslator.java | 8 +-
.../construction/SplittableParDoNaiveBounded.java | 5 +-
.../core/construction/PTransformMatchersTest.java | 3 +-
.../core/construction/SplittableParDoTest.java | 7 +-
runners/core-java/build.gradle | 1 -
...TimeBoundedSplittableProcessElementInvoker.java | 30 ++--
.../core/SplittableParDoViaKeyedWorkItems.java | 17 +--
.../core/SplittableProcessElementInvoker.java | 7 +-
...BoundedSplittableProcessElementInvokerTest.java | 35 +++--
.../runners/core/SplittableParDoProcessFnTest.java | 37 ++---
.../SplittableProcessElementsEvaluatorFactory.java | 23 +--
.../flink/FlinkStreamingTransformTranslators.java | 7 +-
.../wrappers/streaming/SplittableDoFnOperator.java | 4 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 4 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 16 ++-
.../java/org/apache/beam/sdk/transforms/Watch.java | 2 +-
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 2 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 29 +++-
.../splittabledofn/ByteKeyRangeTracker.java| 12 +-
.../splittabledofn/OffsetRangeTracker.java | 10 +-
.../splittabledofn/RestrictionTracker.java | 51 ++-
.../java/org/apache/beam/sdk/io/AvroIOTest.java| 2 -
.../java/org/apache/beam/sdk/io/FileIOTest.java| 2 -
.../org/apache/beam/sdk/io/TextIOReadTest.java | 2 -
.../beam/sdk/transforms/SplittableDoFnTest.java| 22 ++-
.../org/apache/beam/sdk/transforms/WatchTest.java | 9 --
.../sdk/transforms/reflect/DoFnInvokersTest.java | 15 +-
.../reflect/DoFnSignaturesSplittableDoFnTest.java | 46 +++---
.../sdk/fn/splittabledofn/RestrictionTrackers.java | 138 --
.../beam/sdk/fn/splittabledofn/package-info.java | 28
.../fn/splittabledofn/RestrictionTrackersTest.java | 156 -
.../harness/SplittableProcessElementsRunner.java | 11 +-
.../beam/sdk/io/hbase/HBaseReadSplittableDoFn.java | 4 +-
33 files changed, 239 insertions(+), 506 deletions(-)
diff --git
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
index ca1c7ff..c54ad98 100644
---
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
+++
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoTranslator.java
@@ -35,6 +35,7 @@ import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.PCollectionView;
import org.apache.beam.sdk.values.PValue;
@@ -125,12 +126,13 @@ class ParDoTranslator
}
}
- static class SplittableProcessElementsTranslator
- implements TransformTranslator> {
+ static class SplittableProcessElementsTranslator<
+ InputT, OutputT, RestrictionT, TrackerT extends
RestrictionTracker>
+ implements TransformTranslator> {
@Override
public void translate(
-ProcessElements transform,
+ProcessElements transform,
TranslationContext context) {
Map, PValue> outputs = context.getOutputs();
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
index 3197c69..2239743 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/SplittableParDoNaiveBounded.java
@@ -109,7 +109,8 @@ public class SplittableParDoNaiveBounded {
}
}
- static class NaiveProcessFn
+ static class NaiveProcessFn<
+ InputT, OutputT, RestrictionT, TrackerT extends
RestrictionTracker>
[beam] branch master updated (fea721a -> 56babf0)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fea721a Merge pull request #7562 Combiner Lifting Environment new 152b609 release guide changes new db5b086 Change Issue Type to Bug. Look over the flaky label as well. Revert to explicit numbering new c5b1fe7 address readability suggestions new f099a20 address readability suggestions new acad7ab edit wording for flaky tests for the release manager to delegate the fix for flaky tests new 56babf0 [BEAM-6445]: Release Guide changes for release process improvement The 19570 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/contribute/release-guide.md | 124 1 file changed, 79 insertions(+), 45 deletions(-)
[beam] branch master updated (d4bd8bb -> dccd672)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d4bd8bb Merge pull request #7557: [BEAM-2928]Minor code refactoring and improvements. new fbb9aa6 [BEAM-6138] Refactor the start and finish function registration so that PTransform IDs can be properly injected onto user counters. new dccd672 Merge pull request #7482: [BEAM-6138] Refactor the start and finish function registration so that PTransform IDs can be properly injected onto user counters. The 19558 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/fn/harness/BeamFnDataReadRunner.java | 12 ++-- .../beam/fn/harness/BeamFnDataWriteRunner.java | 11 ++- .../beam/fn/harness/BoundedSourceRunner.java | 10 +-- .../org/apache/beam/fn/harness/CombineRunners.java | 12 ++-- .../fn/harness/DoFnPTransformRunnerFactory.java| 15 ++--- .../org/apache/beam/fn/harness/FlattenRunner.java | 8 +-- .../org/apache/beam/fn/harness/MapFnRunners.java | 8 +-- .../beam/fn/harness/PTransformRunnerFactory.java | 11 ++- .../fn/harness/control/ProcessBundleHandler.java | 33 - .../harness/data/PTransformFunctionRegistry.java | 78 ++ .../beam/fn/harness/AssignWindowsRunnerTest.java | 4 +- .../beam/fn/harness/BeamFnDataReadRunnerTest.java | 14 ++-- .../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 14 ++-- .../beam/fn/harness/BoundedSourceRunnerTest.java | 14 ++-- .../apache/beam/fn/harness/CombineRunnersTest.java | 52 +++ .../apache/beam/fn/harness/FlattenRunnerTest.java | 8 +-- .../beam/fn/harness/FnApiDoFnRunnerTest.java | 58 .../apache/beam/fn/harness/MapFnRunnersTest.java | 38 +-- .../harness/control/ProcessBundleHandlerTest.java | 45 +++-- .../data/PTransformFunctionRegistryTest.java | 30 + 20 files changed, 280 insertions(+), 195 deletions(-) create mode 100644 sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/PTransformFunctionRegistry.java copy runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/environment/ProcessEnvironmentTest.java => sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/PTransformFunctionRegistryTest.java (55%)
[beam] branch master updated: Minor code refactoring and improvements.
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9214fa9 Minor code refactoring and improvements.
new d4bd8bb Merge pull request #7557: [BEAM-2928]Minor code refactoring
and improvements.
9214fa9 is described below
commit 9214fa9422800df92689ce784bc81ce062fb3ede
Author: Ruoyun
AuthorDate: Wed Jan 2 12:48:02 2019 -0800
Minor code refactoring and improvements.
---
.../runners/core/construction/graph/SideInputReference.java| 9 +
.../org/apache/beam/runners/direct/CloningBundleFactory.java | 9 +
.../apache/beam/runners/direct/ImmutableListBundleFactory.java | 10 ++
.../org/apache/beam/runners/direct/PCollectionViewWriter.java | 1 +
.../direct/portable/ExecutorServiceParallelExecutor.java | 10 +-
.../beam/runners/direct/portable/PCollectionViewWriter.java| 1 +
.../org/apache/beam/runners/direct/portable/PortableGraph.java | 4
.../apache/beam/runners/direct/portable/ReferenceRunner.java | 6 ++
8 files changed, 41 insertions(+), 9 deletions(-)
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SideInputReference.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SideInputReference.java
index 29e2746..e329743 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SideInputReference.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SideInputReference.java
@@ -18,6 +18,7 @@
package org.apache.beam.runners.core.construction.graph;
import com.google.auto.value.AutoValue;
+import com.google.common.base.MoreObjects;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
@@ -58,4 +59,12 @@ public abstract class SideInputReference {
public abstract String localName();
/** The PCollection that backs this side input. */
public abstract PCollectionNode collection();
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("Transform", transform().toString())
+.add("PCollection", collection().toString())
+.toString();
+ }
}
diff --git
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CloningBundleFactory.java
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CloningBundleFactory.java
index 9c35ef2..25571da 100644
---
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CloningBundleFactory.java
+++
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/CloningBundleFactory.java
@@ -17,6 +17,7 @@
*/
package org.apache.beam.runners.direct;
+import com.google.common.base.MoreObjects;
import org.apache.beam.runners.local.StructuralKey;
import org.apache.beam.sdk.coders.Coder;
import org.apache.beam.sdk.coders.CoderException;
@@ -92,5 +93,13 @@ class CloningBundleFactory implements BundleFactory {
public CommittedBundle commit(Instant synchronizedProcessingTime) {
return underlying.commit(synchronizedProcessingTime);
}
+
+@Override
+public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("Data", underlying.toString())
+ .add("Coder", coder.toString())
+ .toString();
+}
}
}
diff --git
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
index f84043a..58484e1 100644
---
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
+++
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ImmutableListBundleFactory.java
@@ -21,6 +21,7 @@ import static
org.apache.beam.vendor.guava.v20_0.com.google.common.base.Precondi
import static
org.apache.beam.vendor.guava.v20_0.com.google.common.base.Preconditions.checkState;
import com.google.auto.value.AutoValue;
+import com.google.common.base.MoreObjects;
import java.util.Iterator;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
@@ -111,6 +112,15 @@ class ImmutableListBundleFactory implements BundleFactory {
return CommittedImmutableListBundle.create(
pcollection, key, committedElements, minSoFar,
synchronizedCompletionTime);
}
+
+@Override
+public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("Key", key.toString())
+ .add("PCollection", pcollection)
+
[beam] branch master updated: Add all postcommit tests into RC process
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new eb3ad65 Add all postcommit tests into RC process
new ccc9d79 Merge pull request #7509: Add all postcommit tests into RC
process
eb3ad65 is described below
commit eb3ad65e6c12e14a30e7ef7af6e0c9fc78b129ac
Author: Boyuan Zhang
AuthorDate: Mon Jan 14 18:55:17 2019 -0800
Add all postcommit tests into RC process
---
release/src/main/scripts/run_rc_validation.sh| 21 +++
release/src/main/scripts/verify_release_build.sh | 72
2 files changed, 93 insertions(+)
diff --git a/release/src/main/scripts/run_rc_validation.sh
b/release/src/main/scripts/run_rc_validation.sh
index 4881066..1fc0d54 100755
--- a/release/src/main/scripts/run_rc_validation.sh
+++ b/release/src/main/scripts/run_rc_validation.sh
@@ -49,6 +49,8 @@ LOCAL_CLONE_DIR=rc_validations
BEAM_ROOT_DIR=beam
GIT_REPO_URL=https://github.com/apache/beam.git
PYTHON_RC_DOWNLOAD_URL=https://dist.apache.org/repos/dist/dev/beam
+HUB_VERSION=2.5.0
+HUB_ARTIFACTS_NAME=hub-linux-amd64-${HUB_VERSION}
echo "[Input Required] Please enter the release version: "
read RELEASE
@@ -70,6 +72,25 @@ if [[ $confirmation != "y" ]]; then
exit
fi
+echo "=Checking hub"
+if [[ -z `which hub` ]]; then
+ echo "There is no hub installed on your machine."
+ echo "Would you like to install hub with root permission? [y|N]"
+ read confirmation
+ if [[ $confirmation != "y" ]]; then
+echo "Refused to install hub. Cannot proceed into next setp."
+exit
+ fi
+ echo "=Installing hub==="
+ wget
https://github.com/github/hub/releases/download/v${HUB_VERSION}/${HUB_ARTIFACTS_NAME}.tgz
+ tar zvxvf ${HUB_ARTIFACTS_NAME}.tgz
+ sudo ./${HUB_ARTIFACTS_NAME}/install
+ echo "eval "$(hub alias -s)"" >> ~/.bashrc
+ rm -rf ${HUB_ARTIFACTS_NAME}*
+fi
+hub version
+
+
echo "Cloning Beam Release Branch"
cd ~
if [[ -d ${LOCAL_CLONE_DIR} ]]; then
diff --git a/release/src/main/scripts/verify_release_build.sh
b/release/src/main/scripts/verify_release_build.sh
index b284950..50b2c65 100755
--- a/release/src/main/scripts/verify_release_build.sh
+++ b/release/src/main/scripts/verify_release_build.sh
@@ -102,6 +102,27 @@ else
which time
fi
+echo "=Checking hub"
+HUB_VERSION=2.5.0
+HUB_ARTIFACTS_NAME=hub-linux-amd64-${HUB_VERSION}
+if [[ -z `which hub` ]]; then
+ echo "There is no hub installed on your machine."
+ echo "Would you like to install hub with root permission? [y|N]"
+ read confirmation
+ if [[ $confirmation != "y" ]]; then
+echo "Refused to install hub. Cannot proceed into next setp."
+exit
+ fi
+ echo "=Installing hub==="
+ wget
https://github.com/github/hub/releases/download/v${HUB_VERSION}/${HUB_ARTIFACTS_NAME}.tgz
+ tar zvxvf ${HUB_ARTIFACTS_NAME}.tgz
+ sudo ./${HUB_ARTIFACTS_NAME}/install
+ echo "eval "$(hub alias -s)"" >> ~/.bashrc
+ rm -rf ${HUB_ARTIFACTS_NAME}*
+fi
+hub version
+
+cd ~
echo "==Starting Clone Repo=="
if [[ -d ${LOCAL_CLONE_DIR} ]]; then
rm -rf ${LOCAL_CLONE_DIR}
@@ -121,6 +142,57 @@ gpg --output ~/doc.sig --sign ~/.bashrc
./gradlew build -PisRelease --scan --stacktrace --no-parallel --continue
echo "==="
+echo "[Current Task] Run All PostCommit Tests against Release Branch"
+echo "This task will create a PR against apache/beam."
+echo "After PR created, you need to comment phrases listed in description in
the created PR:"
+
+echo "[Confirmation Required] Do you want to proceed? [y|N]"
+read confirmation
+if [[ $confirmation = "y" ]]; then
+ echo "[Input Required] Please enter your github repo URL forked from
apache/beam:"
+ read USER_REMOTE_URL
+ echo "[Input Required] Please enter your github username:"
+ read GITHUB_USERNAME
+ echo "[Input Required] Please enter your github token:"
+ read GITHUB_TOKEN
+ export GITHUB_TOKEN=${GITHUB_TOKEN}
+ WORKING_BRANCH=postcommit_validation_pr
+ git checkout -b ${WORKING_BRANCH}
+ touch empty_file.txt
+ git add empty_file.txt
+ git commit -m "Add empty file in order to create PR"
+ git push -f ${USER_REMOTE_URL}
+ hub pull-request -o -b apache:${branch} -h
${GITHUB_USERNAME}:${WORKING_BRANCH} -F- <<<"[DO NOT MERGE] Run all PostCommit
and PreCommit Tes
[beam] branch master updated: Update syncjenkins shema
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 5e9f33e Update syncjenkins shema new 60b99eb Merge pull request #7541: [BEAM-6456] Update jenkins_builds schema to BIGINT for durations 5e9f33e is described below commit 5e9f33e6d494811607f25662109a72b3ab78d4a0 Author: Mikhail Gryzykhin AuthorDate: Wed Jan 16 12:51:29 2019 -0800 Update syncjenkins shema --- .test-infra/metrics/sync/jenkins/syncjenkins.py | 18 +- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/.test-infra/metrics/sync/jenkins/syncjenkins.py b/.test-infra/metrics/sync/jenkins/syncjenkins.py index 4e9cf74..f89c4b4 100644 --- a/.test-infra/metrics/sync/jenkins/syncjenkins.py +++ b/.test-infra/metrics/sync/jenkins/syncjenkins.py @@ -43,16 +43,16 @@ build_url varchar, build_result varchar, build_timestamp TIMESTAMP, build_builtOn varchar, -build_duration integer, -build_estimatedDuration integer, +build_duration BIGINT, +build_estimatedDuration BIGINT, build_fullDisplayName varchar, -timing_blockedDurationMillis integer, -timing_buildableDurationMillis integer, -timing_buildingDurationMillis integer, -timing_executingTimeMillis integer, -timing_queuingDurationMillis integer, -timing_totalDurationMillis integer, -timing_waitingDurationMillis integer, +timing_blockedDurationMillis BIGINT, +timing_buildableDurationMillis BIGINT, +timing_buildingDurationMillis BIGINT, +timing_executingTimeMillis BIGINT, +timing_queuingDurationMillis BIGINT, +timing_totalDurationMillis BIGINT, +timing_waitingDurationMillis BIGINT, primary key(job_name, build_id) ) """
[beam] 01/01: Merge pull request #7451: Fix typo in comment.
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 84892886a575b7a2484c5ade022961977720f9fd Merge: 336fe0c 07d0a95 Author: Scott Wegner AuthorDate: Thu Jan 10 12:07:16 2019 -0800 Merge pull request #7451: Fix typo in comment. .../beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (336fe0c -> 8489288)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 336fe0c Merge pull request #7457: [BEAM-6294] Ensure input and output coders are equal for reshuffle transforms. add 07d0a95 Fix typo in comment. new 8489288 Merge pull request #7451: Fix typo in comment. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (bacec1f -> 3907e01)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bacec1f Merge pull request #7443: [BEAM-6382] SamzaRunner: add an option to read configs using a user-defined factory add 90a33d2 [BEAM-6349] & [BEAM-6368] Build worker and use it when running load tests on Dataflow new 3907e01 Merge pull request #7435: [BEAM-6349] & [BEAM-6368] Build worker and use it when running loadtests on Dataflow The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/testing/load-tests/build.gradle | 28 +--- 1 file changed, 25 insertions(+), 3 deletions(-)
[beam] 01/01: Merge pull request #7435: [BEAM-6349] & [BEAM-6368] Build worker and use it when running loadtests on Dataflow
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 3907e01de61144a9e64aa49229bf505bd520774c Merge: bacec1f 90a33d2 Author: Scott Wegner AuthorDate: Wed Jan 9 09:53:16 2019 -0800 Merge pull request #7435: [BEAM-6349] & [BEAM-6368] Build worker and use it when running loadtests on Dataflow sdks/java/testing/load-tests/build.gradle | 28 +--- 1 file changed, 25 insertions(+), 3 deletions(-)
[beam] 01/01: Merge pull request #7421: Change warning logic to only consider work items without an ID to be invalid. Not the absence of a work item.
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a4e30835976aa8f7c2b5a076f3766b8227d6f951 Merge: 24f83f0 717ddba Author: Scott Wegner AuthorDate: Mon Jan 7 09:09:00 2019 -0800 Merge pull request #7421: Change warning logic to only consider work items without an ID to be invalid. Not the absence of a work item. .../dataflow/worker/DataflowWorkUnitClient.java| 7 +- .../worker/DataflowWorkUnitClientTest.java | 26 +- 2 files changed, 31 insertions(+), 2 deletions(-)
[beam] branch master updated (24f83f0 -> a4e3083)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 24f83f0 Merge pull request #7407: Fix Dataflow Java Portable VR configuration add 717ddba Change warning logic to only consider work items without an ID to be invalid. Not the absence of a work item. new a4e3083 Merge pull request #7421: Change warning logic to only consider work items without an ID to be invalid. Not the absence of a work item. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../dataflow/worker/DataflowWorkUnitClient.java| 7 +- .../worker/DataflowWorkUnitClientTest.java | 26 +- 2 files changed, 31 insertions(+), 2 deletions(-)
[beam] 01/01: Merge pull request #7407: Fix Dataflow Java Portable VR configuration
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 24f83f096f28d902257ccdc3c8ffcdc4f5cee5a7 Merge: 28353e8 f43ea10 Author: Scott Wegner AuthorDate: Mon Jan 7 09:06:40 2019 -0800 Merge pull request #7407: Fix Dataflow Java Portable VR configuration ...b_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy | 2 +- runners/google-cloud-dataflow-java/build.gradle | 6 -- 2 files changed, 1 insertion(+), 7 deletions(-)
[beam] branch master updated (28353e8 -> 24f83f0)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 28353e8 Merge pull request #7387: [BEAM-6378] Updating Tika add f43ea10 Fix Portable VR configuration new 24f83f0 Merge pull request #7407: Fix Dataflow Java Portable VR configuration The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: ...b_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy | 2 +- runners/google-cloud-dataflow-java/build.gradle | 6 -- 2 files changed, 1 insertion(+), 7 deletions(-)
[beam] branch master updated (8a7f971 -> a0bc8a4)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8a7f971 [BEAM-6362] remove --info from gradle invocations add f22d12d Reduce days to keep Jenkins job logs to 14 new a0bc8a4 Merge pull request #7410: Reduce days to keep Jenkins job logs to 14 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .test-infra/jenkins/CommonJobProperties.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #7410: Reduce days to keep Jenkins job logs to 14
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a0bc8a458261a4d49f69ae084643454417cd1da9 Merge: 8a7f971 f22d12d Author: Scott Wegner AuthorDate: Fri Jan 4 13:42:05 2019 -0800 Merge pull request #7410: Reduce days to keep Jenkins job logs to 14 .test-infra/jenkins/CommonJobProperties.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #7401: Upgrade Gradle to 4.10.3
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 20abb3ef5d73894069657d6670d371bd435c5a8c Merge: bd80118 ea0b8d5 Author: Scott Wegner AuthorDate: Fri Jan 4 08:09:36 2019 -0800 Merge pull request #7401: Upgrade Gradle to 4.10.3 gradle/wrapper/gradle-wrapper.jar| Bin 56177 -> 56177 bytes gradle/wrapper/gradle-wrapper.properties | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (bd80118 -> 20abb3e)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bd80118 Merge pull request #7400 from echauchot/BEAM-6030-metrics-sinks-pipelineOptions add ea0b8d5 Upgrade Gradle to 4.10.3 new 20abb3e Merge pull request #7401: Upgrade Gradle to 4.10.3 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: gradle/wrapper/gradle-wrapper.jar| Bin 56177 -> 56177 bytes gradle/wrapper/gradle-wrapper.properties | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #7393: [BEAM-5662] Clean up website html-proofer config
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a2986cc194a27bee80706b8d282c1ee912520395 Merge: c148c35 5466ac0 Author: Scott Wegner AuthorDate: Thu Jan 3 09:00:16 2019 -0800 Merge pull request #7393: [BEAM-5662] Clean up website html-proofer config website/Gemfile.lock | 16 website/Rakefile | 13 + website/src/_posts/2017-01-09-added-apex-runner.md | 4 ++-- website/src/documentation/io/built-in-google-bigquery.md | 2 +- website/src/documentation/runners/apex.md| 2 +- website/src/documentation/sdks/euphoria.md | 4 ++-- 6 files changed, 19 insertions(+), 22 deletions(-)
[beam] branch master updated (c148c35 -> a2986cc)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c148c35 Merge pull request #7390: [BEAM-6339] Add paddedcell fix to spotlessJava rules. add c028ebc Upgrade html-proofer and dependencies to latest add 07c279a Remove broken links to datatorrent.com add b09e721 Fix pydoc link to GoogleCloudOptions add fd5e321 Remove broken link to atrato.io add a79ef89 Fix link to internal anchor add 5466ac0 Remove stale exclusions from HTML link checker. new a2986cc Merge pull request #7393: [BEAM-5662] Clean up website html-proofer config The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/Gemfile.lock | 16 website/Rakefile | 13 + website/src/_posts/2017-01-09-added-apex-runner.md | 4 ++-- website/src/documentation/io/built-in-google-bigquery.md | 2 +- website/src/documentation/runners/apex.md| 2 +- website/src/documentation/sdks/euphoria.md | 4 ++-- 6 files changed, 19 insertions(+), 22 deletions(-)
[beam] 01/01: Merge pull request #7390: [BEAM-6339] Add paddedcell fix to spotlessJava rules.
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit c148c35dbe3394e000b34d0d32596a9d629d51d3 Merge: 5130bcb a404cee Author: Scott Wegner AuthorDate: Thu Jan 3 08:46:33 2019 -0800 Merge pull request #7390: [BEAM-6339] Add paddedcell fix to spotlessJava rules. .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-)
[beam] branch master updated (5130bcb -> c148c35)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5130bcb Merge pull request #7385 from echauchot/exposeSerializationSerializablePipelineOptions add a404cee Add paddedcell fix to spotlessJava rules. new c148c35 Merge pull request #7390: [BEAM-6339] Add paddedcell fix to spotlessJava rules. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 8 +--- 1 file changed, 5 insertions(+), 3 deletions(-)
[beam] branch master updated (15aa88d -> a25b64d)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 15aa88d Merge pull request #7328: [BEAM-6056] Upgrade vendored gRPC artifact version to 0.2 add 14781c7 [BEAM-6056] Source vendored grpc dependency from Maven central new a25b64d Merge pull request #7388: [BEAM-6056] Source vendored grpc dependency from Maven central The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 3 ++- runners/core-construction-java/build.gradle | 2 +- runners/direct-java/build.gradle | 2 +- runners/flink/flink_runner.gradle | 2 +- runners/google-cloud-dataflow-java/build.gradle | 2 +- runners/google-cloud-dataflow-java/worker/build.gradle| 2 +- runners/google-cloud-dataflow-java/worker/legacy-worker/build.gradle | 2 +- runners/java-fn-execution/build.gradle| 2 +- runners/reference/java/build.gradle | 2 +- sdks/java/fn-execution/build.gradle | 2 +- sdks/java/harness/build.gradle| 4 ++-- 11 files changed, 13 insertions(+), 12 deletions(-)
[beam] 01/01: Merge pull request #7388: [BEAM-6056] Source vendored grpc dependency from Maven central
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a25b64d45072270a4df84b74aa58d4b492e05382 Merge: 15aa88d 14781c7 Author: Scott Wegner AuthorDate: Wed Jan 2 10:29:45 2019 -0800 Merge pull request #7388: [BEAM-6056] Source vendored grpc dependency from Maven central .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 3 ++- runners/core-construction-java/build.gradle | 2 +- runners/direct-java/build.gradle | 2 +- runners/flink/flink_runner.gradle | 2 +- runners/google-cloud-dataflow-java/build.gradle | 2 +- runners/google-cloud-dataflow-java/worker/build.gradle| 2 +- runners/google-cloud-dataflow-java/worker/legacy-worker/build.gradle | 2 +- runners/java-fn-execution/build.gradle| 2 +- runners/reference/java/build.gradle | 2 +- sdks/java/fn-execution/build.gradle | 2 +- sdks/java/harness/build.gradle| 4 ++-- 11 files changed, 13 insertions(+), 12 deletions(-)
[beam] branch master updated (926361b -> 15aa88d)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 926361b [BEAM-5386] Move assertion out of finally block to not swallow original exception add 3b8abca Upgrade vendored gRPC artifact version to 0.2 add 15aa88d Merge pull request #7328: [BEAM-6056] Upgrade vendored gRPC artifact version to 0.2 No new revisions were added by this update. Summary of changes: vendor/grpc-1_13_1/build.gradle | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (f036699 -> a5a139d)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f036699 Merge pull request #7320: [BEAM-6270] Fix documentation for PCollection Type in XmlIO example add c0655d1 Rename v1_13_1 to v1p13p1. add 85ea7ba Depend on local vendored jar instead of the one from Maven add 2afc42b Disable 'jar' task for vendoring projects. add a5a139d Merge pull request #7324: [BEAM-6056] Rename vendored guava relocation to v1p13p1 No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 5 ++-- .../org/apache/beam/gradle/GrpcVendoring.groovy| 8 +++--- .../org/apache/beam/gradle/VendorJavaPlugin.groovy | 4 +++ runners/core-construction-java/build.gradle| 2 +- .../core/construction/ArtifactServiceStager.java | 6 ++-- .../beam/runners/core/construction/BeamUrns.java | 2 +- .../core/construction/CoderTranslation.java| 2 +- .../core/construction/CombineTranslation.java | 2 +- .../CreatePCollectionViewTranslation.java | 2 +- .../core/construction/DisplayDataTranslation.java | 4 +-- .../runners/core/construction/Environments.java| 4 +-- .../construction/PCollectionViewTranslation.java | 2 +- .../core/construction/ParDoTranslation.java| 4 +-- .../construction/PipelineOptionsTranslation.java | 6 ++-- .../runners/core/construction/ReadTranslation.java | 4 +-- .../core/construction/TestStreamTranslation.java | 2 +- .../core/construction/WindowIntoTranslation.java | 2 +- .../construction/WindowingStrategyTranslation.java | 8 +++--- .../core/construction/WriteFilesTranslation.java | 2 +- .../graph/GreedyPCollectionFusers.java | 2 +- .../core/construction/graph/QueryablePipeline.java | 2 +- .../construction/ArtifactServiceStagerTest.java| 6 ++-- .../InMemoryArtifactStagerService.java | 2 +- .../PipelineOptionsTranslationTest.java| 6 ++-- .../construction/WindowIntoTranslationTest.java| 2 +- .../construction/graph/ProtoOverridesTest.java | 2 +- runners/direct-java/build.gradle | 2 +- .../runners/direct/portable/ReferenceRunner.java | 2 +- .../LocalFileSystemArtifactRetrievalService.java | 6 ++-- .../LocalFileSystemArtifactStagerService.java | 8 +++--- .../runners/direct/portable/job/PreparingJob.java | 2 +- .../portable/job/ReferenceRunnerJobService.java| 6 ++-- ...ocalFileSystemArtifactRetrievalServiceTest.java | 4 +-- .../LocalFileSystemArtifactStagerServiceTest.java | 10 +++ .../UnsupportedArtifactRetrievalServiceTest.java | 2 +- .../job/ReferenceRunnerJobServiceTest.java | 4 +-- runners/flink/flink_runner.gradle | 2 +- .../FlinkBatchPortablePipelineTranslator.java | 2 +- .../apache/beam/runners/flink/FlinkJobInvoker.java | 2 +- .../FlinkStreamingPortablePipelineTranslator.java | 2 +- .../utils/FlinkPipelineTranslatorUtils.java| 2 +- .../streaming/ExecutableStageDoFnOperatorTest.java | 2 +- .../FlinkDefaultExecutableStageContextTest.java| 2 +- .../FlinkExecutableStageFunctionTest.java | 2 +- runners/google-cloud-dataflow-java/build.gradle| 2 +- .../dataflow/DataflowPipelineTranslator.java | 2 +- .../google-cloud-dataflow-java/worker/build.gradle | 2 +- .../worker/legacy-worker/build.gradle | 2 +- .../runners/dataflow/worker/ByteStringCoder.java | 2 +- .../dataflow/worker/DataflowRunnerHarness.java | 2 +- .../worker/DataflowWorkerHarnessHelper.java| 2 +- .../worker/GroupAlsoByWindowParDoFnFactory.java| 2 +- .../worker/MetricTrackingWindmillServerStub.java | 2 +- .../beam/runners/dataflow/worker/PubsubSink.java | 2 +- .../beam/runners/dataflow/worker/ReaderCache.java | 2 +- .../beam/runners/dataflow/worker/StateFetcher.java | 2 +- .../dataflow/worker/StreamingDataflowWorker.java | 2 +- .../worker/StreamingModeExecutionContext.java | 2 +- .../dataflow/worker/StreamingSideInputFetcher.java | 4 +-- .../dataflow/worker/WindmillNamespacePrefix.java | 2 +- .../beam/runners/dataflow/worker/WindmillSink.java | 2 +- .../dataflow/worker/WindmillStateCache.java| 2 +- .../dataflow/worker/WindmillStateInternals.java| 2 +- .../dataflow/worker/WindmillStateReader.java | 2 +- .../dataflow/worker/WindmillTimerInternals.java| 2 +- .../dataflow/worker/WorkerCustomSources.java | 2 +- .../dataflow/worker/fn/BeamFnControlService.java | 2 +- .../control/RegisterAndProcessBundleOperation.java | 4 +-- .../worker/fn/data/BeamFnDataGrpcService.java | 2 +- .../worker/fn/logging/BeamFnLoggingService.java| 4 +-- .../fn/stream/ServerStreamObserverFactory.java | 6 ++-- .../graph/CreateExecutableStageNodeFunction.j
[beam] branch master updated (0c589db -> f036699)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0c589db [BEAM-5993] Create SideInput Load test (#7020) add 61d7e53 Fix documentation for PCollection Type in XMLIO example add f036699 Merge pull request #7320: [BEAM-6270] Fix documentation for PCollection Type in XmlIO example No new revisions were added by this update. Summary of changes: sdks/java/io/xml/src/main/java/org/apache/beam/sdk/io/xml/XmlIO.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (85bd95e -> bd68ef6)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 85bd95e Merge pull request #7296: [BEAM-6245] Add integration test for FlinkTransformOverrides add 8214cb6 [BEAM-5449] Tagging failing ULR ValidatesRunner tests. new bd68ef6 Merge pull request #7295: [BEAM-5449] Tagging failing ULR ValidatesRunner tests. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/direct-java/build.gradle | 22 ++ .../org/apache/beam/sdk/io/CountingSourceTest.java | 13 +- .../org/apache/beam/sdk/testing/PAssertTest.java | 6 ++- .../apache/beam/sdk/transforms/CombineFnsTest.java | 5 ++- .../apache/beam/sdk/transforms/CombineTest.java| 48 ++ .../org/apache/beam/sdk/transforms/ViewTest.java | 2 + 6 files changed, 76 insertions(+), 20 deletions(-)
[beam] 01/01: Merge pull request #7295: [BEAM-5449] Tagging failing ULR ValidatesRunner tests.
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit bd68ef6dab4feea9f02416f9af2b59d834814064 Merge: 85bd95e 8214cb6 Author: Scott Wegner AuthorDate: Wed Dec 19 08:45:11 2018 -0800 Merge pull request #7295: [BEAM-5449] Tagging failing ULR ValidatesRunner tests. runners/direct-java/build.gradle | 22 ++ .../org/apache/beam/sdk/io/CountingSourceTest.java | 13 +- .../org/apache/beam/sdk/testing/PAssertTest.java | 6 ++- .../apache/beam/sdk/transforms/CombineFnsTest.java | 5 ++- .../apache/beam/sdk/transforms/CombineTest.java| 48 ++ .../org/apache/beam/sdk/transforms/ViewTest.java | 2 + 6 files changed, 76 insertions(+), 20 deletions(-)
[beam] 01/01: Merge pull request #7311: Update release guide with new Jenkins job name.
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit aa0fdc5ce60c61ee10ded4796ef6aabe248dc9ec Merge: e4577ed 0ef5e66 Author: Scott Wegner AuthorDate: Tue Dec 18 15:55:22 2018 -0800 Merge pull request #7311: Update release guide with new Jenkins job name. website/src/contribute/release-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (e4577ed -> aa0fdc5)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e4577ed Merge pull request #7257 Fix time format on Windows. add 0ef5e66 Update release guide with new Jenkins job name. new aa0fdc5 Merge pull request #7311: Update release guide with new Jenkins job name. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/src/contribute/release-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated: [BEAM-6225] Setup Jenkins Job to Run VR with ExecutableStage (#7271)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b7035c1 [BEAM-6225] Setup Jenkins Job to Run VR with ExecutableStage
(#7271)
b7035c1 is described below
commit b7035c1a098c526356c6fb33480b989ed037da0a
Author: Boyuan Zhang <36090911+boyua...@users.noreply.github.com>
AuthorDate: Fri Dec 14 13:47:48 2018 -0800
[BEAM-6225] Setup Jenkins Job to Run VR with ExecutableStage (#7271)
---
...unner_DataflowPortabilityExecutableStage.groovy | 54 ++
runners/google-cloud-dataflow-java/build.gradle| 48 +--
...aflowPortabilityExecutableStageUnsupported.java | 25 ++
.../apache/beam/sdk/testing/UsesSideInputs.java| 24 ++
.../org/apache/beam/sdk/testing/PAssertTest.java | 12 ++---
.../apache/beam/sdk/transforms/CombineTest.java| 30 ++--
.../apache/beam/sdk/transforms/FlattenTest.java| 3 +-
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 11 +++--
.../org/apache/beam/sdk/transforms/ParDoTest.java | 17 +++
.../apache/beam/sdk/transforms/ReshuffleTest.java | 11 +++--
.../beam/sdk/transforms/SplittableDoFnTest.java| 10 +++-
.../beam/sdk/transforms/join/CoGroupByKeyTest.java | 8 ++--
.../beam/sdk/transforms/windowing/WindowTest.java | 5 +-
.../sdk/transforms/windowing/WindowingTest.java| 9 ++--
14 files changed, 212 insertions(+), 55 deletions(-)
diff --git
a/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy
b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy
new file mode 100644
index 000..62e7361
--- /dev/null
+++
b/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+
+// This job runs the suite of ValidatesRunner tests against the Dataflow
+// runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage',
+ 'Run Dataflow Portability ExecutableStage ValidatesRunner', 'Google Cloud
Dataflow Runner PortabilityApi ExecutableStage ValidatesRunner Tests', this) {
+
+ description('Runs the ValidatesRunner suite on the Dataflow PortabilityApi
runner with ExecutableStage code path enabled.')
+
+ // Set common parameters. Sets a 3 hour timeout.
+ commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 400)
+
+ // Publish all test results to Jenkins
+ publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+ }
+
+ // Gradle goals for this job.
+ steps {
+gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+
tasks(':beam-runners-google-cloud-dataflow-java:validatesRunnerFnApiWorkerExecutableStageTest')
+ // Increase parallel worker threads above processor limit since most
time is
+ // spent waiting on Dataflow jobs. ValidatesRunner tests on Dataflow are
slow
+ // because each one launches a Dataflow job with about 3 mins of
overhead.
+ // 3 x num_cores strikes a good balance between maxing out parallelism
without
+ // overloading the machines.
+ commonJobProperties.setGradleSwitches(delegate, 3 *
Runtime.runtime.availableProcessors())
+}
+ }
+
+ // [BEAM-6236] "use_executable_stage_bundle_execution" hasn't been rolled
out.
+ disabled()
+}
diff --git a/runners/google-cloud-dataflow-java/build.gradle
b/runners/google-cloud-dataflow-java/build.gradle
index c0b831c..9c6aaf4 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -241,17 +241,55 @@ task validatesRunnerFnApiWorkerTest(type: Test) {
}
}
+task validatesRunnerFnApiWorkerExecutableStageTest(type: Test) {
+group = "Verification"
+description &qu
[beam] branch master updated (5ec695b -> ec3f792)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 5ec695b [BEAM-6150] Superinterface for SerializableFunction allowing
declared exceptions (#7160)
add 4f90294 Remove Gradle from Jenkins job names
new ec3f792 Merge pull request #7286 from swegner/jenkins_gradle
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md| 4 ++--
...ob_PostCommit_Go_GradleBuild.groovy => job_PostCommit_Go.groovy} | 3 ++-
...ostCommit_Java_GradleBuild.groovy => job_PostCommit_Java.groovy} | 3 ++-
...GradleBuild.groovy => job_PostCommit_Java_PortabilityApi.groovy} | 3 ++-
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Apex.groovy | 5 ++---
.../jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy | 5 ++---
.../jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy| 3 ++-
.../jenkins/job_PostCommit_Java_ValidatesRunner_Gearpump.groovy | 5 ++---
...b_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy | 3 ++-
.../jenkins/job_PostCommit_Java_ValidatesRunner_Samza.groovy| 3 ++-
.../jenkins/job_PostCommit_Java_ValidatesRunner_Spark.groovy| 5 ++---
...le_NightlySnapshot.groovy => job_Release_NightlySnapshot.groovy} | 3 ++-
.test-infra/metrics/dashboards/stability_critical_jobs_status.json | 4 ++--
README.md | 6 +++---
14 files changed, 29 insertions(+), 26 deletions(-)
rename .test-infra/jenkins/{job_PostCommit_Go_GradleBuild.groovy =>
job_PostCommit_Go.groovy} (91%)
rename .test-infra/jenkins/{job_PostCommit_Java_GradleBuild.groovy =>
job_PostCommit_Java.groovy} (92%)
rename
.test-infra/jenkins/{job_PostCommit_Java_PortabilityApi_GradleBuild.groovy =>
job_PostCommit_Java_PortabilityApi.groovy} (94%)
rename .test-infra/jenkins/{job_Release_Gradle_NightlySnapshot.groovy =>
job_Release_NightlySnapshot.groovy} (96%)
[beam] 01/01: Merge pull request #7286 from swegner/jenkins_gradle
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit ec3f79214e9ef204fa32b744051a291fe4b61e23 Merge: 5ec695b 4f90294 Author: Scott Wegner AuthorDate: Fri Dec 14 13:24:35 2018 -0800 Merge pull request #7286 from swegner/jenkins_gradle Remove Gradle from Jenkins job names .github/PULL_REQUEST_TEMPLATE.md| 4 ++-- ...ob_PostCommit_Go_GradleBuild.groovy => job_PostCommit_Go.groovy} | 3 ++- ...ostCommit_Java_GradleBuild.groovy => job_PostCommit_Java.groovy} | 3 ++- ...GradleBuild.groovy => job_PostCommit_Java_PortabilityApi.groovy} | 3 ++- .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Apex.groovy | 5 ++--- .../jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow.groovy | 5 ++--- .../jenkins/job_PostCommit_Java_ValidatesRunner_Flink.groovy| 3 ++- .../jenkins/job_PostCommit_Java_ValidatesRunner_Gearpump.groovy | 5 ++--- ...b_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy | 3 ++- .../jenkins/job_PostCommit_Java_ValidatesRunner_Samza.groovy| 3 ++- .../jenkins/job_PostCommit_Java_ValidatesRunner_Spark.groovy| 5 ++--- ...le_NightlySnapshot.groovy => job_Release_NightlySnapshot.groovy} | 3 ++- .test-infra/metrics/dashboards/stability_critical_jobs_status.json | 4 ++-- README.md | 6 +++--- 14 files changed, 29 insertions(+), 26 deletions(-)
[beam] branch master updated (54e2fc1 -> 977080f)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 54e2fc1 [BEAM-6206] Add CustomHttpErrors a tool to allow adding custom errors for specific failing http calls. Plus, add a custom error message in BigQueryServicesImpl. (#7270) add 69358c5 [BEAM-6191] Remove redundant error logging for Dataflow exception handling new 977080f Merge pull request #7220: [BEAM-6191] Remove redundant error logging for Dataflow exception handling The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/dataflow/worker/WorkItemStatusClient.java | 9 ++--- .../dataflow/worker/util/common/worker/MapTaskExecutor.java | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-)
[beam] 01/01: Merge pull request #7220: [BEAM-6191] Remove redundant error logging for Dataflow exception handling
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 977080f908ac1a8d27d6aab3f4df5550062221ad Merge: 54e2fc1 69358c5 Author: Scott Wegner AuthorDate: Fri Dec 14 09:44:21 2018 -0800 Merge pull request #7220: [BEAM-6191] Remove redundant error logging for Dataflow exception handling .../beam/runners/dataflow/worker/WorkItemStatusClient.java | 9 ++--- .../dataflow/worker/util/common/worker/MapTaskExecutor.java | 2 +- 2 files changed, 7 insertions(+), 4 deletions(-)
[beam] branch master updated: [BEAM-6206] Add CustomHttpErrors a tool to allow adding custom errors for specific failing http calls. Plus, add a custom error message in BigQueryServicesImpl. (#7270)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 54e2fc1 [BEAM-6206] Add CustomHttpErrors a tool to allow adding
custom errors for specific failing http calls. Plus, add a custom error message
in BigQueryServicesImpl. (#7270)
54e2fc1 is described below
commit 54e2fc12ad6c07c43782d03fd95241934b36bda6
Author: Alex Amato
AuthorDate: Fri Dec 14 09:30:48 2018 -0800
[BEAM-6206] Add CustomHttpErrors a tool to allow adding custom errors for
specific failing http calls. Plus, add a custom error message in
BigQueryServicesImpl. (#7270)
---
.../org/apache/beam/sdk/util/CustomHttpErrors.java | 141 +
.../apache/beam/sdk/util/HttpCallCustomError.java | 25
.../org/apache/beam/sdk/util/HttpCallMatcher.java | 28
.../apache/beam/sdk/util/HttpRequestWrapper.java | 40 ++
.../apache/beam/sdk/util/HttpResponseWrapper.java | 38 ++
.../beam/sdk/util/RetryHttpRequestInitializer.java | 32 -
.../apache/beam/sdk/util/CustomHttpErrorsTest.java | 128 +++
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 16 +++
8 files changed, 443 insertions(+), 5 deletions(-)
diff --git
a/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/CustomHttpErrors.java
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/CustomHttpErrors.java
new file mode 100644
index 000..db46d98
--- /dev/null
+++
b/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/util/CustomHttpErrors.java
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * An optional component to use with the {@code RetryHttpRequestInitializer}
in order to provide
+ * custom errors for failing http calls. This class allows you to specify
custom error messages
+ * which match specific error codes and containing strings in the URL. The
first matcher to match
+ * the request and response will be used to provide the custom error.
+ *
+ * The intended use case here is to examine one of the logs emitted by a
failing call made by the
+ * RetryHttpRequestInitializer, and then adding a custom error message which
matches the URL and
+ * code for it.
+ *
+ * Usage: See more in CustomHttpErrorsTest.
+ *
+ * {@code
+ * CustomHttpErrors.Builder builder = new CustomHttpErrors.Builder();
+ * builder.addErrorForCodeAndUrlContains(403,"/tables?", "Custom Error Msg");
+ * CustomHttpErrors customErrors = builder.build();
+ *
+ *
+ * RetryHttpRequestInitializer initializer = ...
+ * initializer.setCustomErrors(customErrors);
+ * }
+ *
+ * Suggestions for future enhancements to anyone upgrading this file:
+ *
+ *
+ * This class is left open for extension, to allow different functions
for HttpCallMatcher and
+ * HttpCallCustomError to match and log errors. For example, new
functionality may include
+ * matching an error based on the HttpResponse body. Additionally,
extracting and logging
+ * strings from the HttpResponse body may make useful functionality.
+ * Add a methods to add custom errors based on inspecting the contents
of the HttpRequest and
+ * HttpResponse
+ * Be sure to update the HttpRequestWrapper and HttpResponseWrapper with
any new getters that
+ * you may use. The wrappers were introduced to add a layer of
indirection which could be
+ * mocked mocked out in tests. This was unfortunately needed because
mockito cannot mock final
+ * classes and its non trivial to just construct HttpRequest and
HttpResponse objects.
+ * Making matchers composable with an AND operator may simplify
enhancing this code, if
+ * several different matchers are used.
+ *
+ *
+ *
+ */
+public class CustomHttpErrors {
+
+ /**
+ * A simple Tuple class for creating a list of HttpRespons
[beam] branch master updated: [BEAM-6138] Add User Counter Metric Support to Java SDK (#6799)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 0b3b9e0 [BEAM-6138] Add User Counter Metric Support to Java SDK
(#6799)
0b3b9e0 is described below
commit 0b3b9e0b99ad798836b789b181b51d47cc5234a6
Author: Alex Amato
AuthorDate: Thu Dec 13 10:10:12 2018 -0800
[BEAM-6138] Add User Counter Metric Support to Java SDK (#6799)
---
.../runners/core/metrics/MetricsContainerImpl.java | 19 +++
.../fnexecution/control/RemoteExecutionTest.java | 129 +++
.../fn/harness/control/ProcessBundleHandler.java | 38 --
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 136 +
4 files changed, 309 insertions(+), 13 deletions(-)
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
index 95bfa74..9147919 100644
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
@@ -21,8 +21,10 @@ import static
com.google.common.base.Preconditions.checkNotNull;
import com.google.common.collect.ImmutableList;
import java.io.Serializable;
+import java.util.ArrayList;
import java.util.Map;
import javax.annotation.Nullable;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
import org.apache.beam.runners.core.construction.metrics.MetricKey;
import org.apache.beam.runners.core.metrics.MetricUpdates.MetricUpdate;
import org.apache.beam.sdk.annotations.Experimental;
@@ -136,6 +138,23 @@ public class MetricsContainerImpl implements Serializable,
MetricsContainer {
extractUpdates(counters), extractUpdates(distributions),
extractUpdates(gauges));
}
+ /** Return the cumulative values for any metrics in this container as
MonitoringInfos. */
+ public Iterable getMonitoringInfos() {
+// Extract user metrics and store as MonitoringInfos.
+ArrayList monitoringInfos = new
ArrayList();
+MetricUpdates mus = this.getUpdates();
+
+for (MetricUpdate mu : mus.counterUpdates()) {
+ SimpleMonitoringInfoBuilder builder = new
SimpleMonitoringInfoBuilder(true);
+ builder.setUrnForUserMetric(
+ mu.getKey().metricName().getNamespace(),
mu.getKey().metricName().getName());
+ builder.setInt64Value(mu.getUpdate());
+ builder.setTimestampToNow();
+ monitoringInfos.add(builder.build());
+}
+return monitoringInfos;
+ }
+
private void commitUpdates(MetricsMap>
cells) {
for (MetricCell cell : cells.values()) {
cell.getDirty().afterCommit();
diff --git
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
index f53257f..1fdf6db 100644
---
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
+++
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
@@ -50,12 +50,16 @@ import java.util.concurrent.Future;
import java.util.concurrent.ThreadFactory;
import java.util.function.Function;
import org.apache.beam.fn.harness.FnHarness;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.MonitoringInfo;
+import
org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleProgressResponse;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleResponse;
import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
import org.apache.beam.model.pipeline.v1.RunnerApi;
import org.apache.beam.runners.core.construction.PipelineTranslation;
import org.apache.beam.runners.core.construction.graph.ExecutableStage;
import org.apache.beam.runners.core.construction.graph.FusedPipeline;
import org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser;
+import org.apache.beam.runners.core.metrics.SimpleMonitoringInfoBuilder;
import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
import org.apache.beam.runners.fnexecution.GrpcFnServer;
import org.apache.beam.runners.fnexecution.InProcessServerFactory;
@@ -83,6 +87,7 @@ import org.apache.beam.sdk.coders.VoidCoder;
import org.apache.beam.sdk.fn.data.FnDataReceiver;
import org.apache.beam.sdk.fn.stream.OutboundObserverFactory;
import org.apache.beam.sdk.fn.test.InProcessManagedChannelFactory;
+import org.apache.beam.sdk.metrics.Metrics;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.state.BagState;
import org.apache.beam.sdk.state.ReadableState;
@@ -109,16 +114,20 @@ imp
[beam] branch master updated (1518361 -> 4eb7744)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1518361 Merge pull request #7174: [BEAM-5978] Changing parallelism for wordcount to 1 add a89d296 Fix broken beam-sdks-python:test new 4eb7744 Merge pull request #7273: Fix broken beam-sdks-python:test The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #7273: Fix broken beam-sdks-python:test
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 4eb774447c1dcce9987921c3178041e92bc0270f Merge: 1518361 a89d296 Author: Scott Wegner AuthorDate: Thu Dec 13 09:40:28 2018 -0800 Merge pull request #7273: Fix broken beam-sdks-python:test buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #7265: Fixing publishing problem introduced in #7197
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 20df151f8128110689df6248291805890139794c Merge: dc10f75 41719ac Author: Scott Wegner AuthorDate: Wed Dec 12 12:50:19 2018 -0800 Merge pull request #7265: Fixing publishing problem introduced in #7197 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (dc10f75 -> 20df151)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from dc10f75 Merge pull request #7244: [BEAM-6138] Add a MonitoringInfoSpec proto and SimpleMonitoringInfoBuilder to pro… add 41719ac Fixing publishing problem introduced in #7197 new 20df151 Merge pull request #7265: Fixing publishing problem introduced in #7197 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #7244: [BEAM-6138] Add a MonitoringInfoSpec proto and SimpleMonitoringInfoBuilder to pro…
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit dc10f75a7bda91f251d43309a21327fc2ab26154 Merge: a61f2c5 11513c3 Author: Scott Wegner AuthorDate: Wed Dec 12 10:44:28 2018 -0800 Merge pull request #7244: [BEAM-6138] Add a MonitoringInfoSpec proto and SimpleMonitoringInfoBuilder to pro… .../fn-execution/src/main/proto/beam_fn_api.proto | 269 ++--- .../core/metrics/SimpleMonitoringInfoBuilder.java | 219 + .../metrics/SimpleMonitoringInfoBuilderTest.java | 87 +++ 3 files changed, 482 insertions(+), 93 deletions(-)
[beam] branch master updated (a61f2c5 -> dc10f75)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a61f2c5 Merge pull request #7242: Add a MonitoringInfoLabelProps proto to attach the proper key string name to MonitoringInfo label Enums add c314dfe Add a MonitoringInfoSpec proto and SimpleMonitoringInfoBuilder to provide specs and validate MonitoringInfos are properly populated. add 11513c3 Merge remote-tracking branch 'upstream/master' into pr7244 new dc10f75 Merge pull request #7244: [BEAM-6138] Add a MonitoringInfoSpec proto and SimpleMonitoringInfoBuilder to pro… The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../fn-execution/src/main/proto/beam_fn_api.proto | 269 ++--- .../core/metrics/SimpleMonitoringInfoBuilder.java | 219 + .../metrics/SimpleMonitoringInfoBuilderTest.java | 87 +++ 3 files changed, 482 insertions(+), 93 deletions(-) create mode 100644 runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilder.java create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/SimpleMonitoringInfoBuilderTest.java
[beam] 01/01: Merge pull request #7242: Add a MonitoringInfoLabelProps proto to attach the proper key string name to MonitoringInfo label Enums
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit a61f2c59076eca6777c995798de9bbc207c780f1 Merge: 5574f47 8828b16 Author: Scott Wegner AuthorDate: Wed Dec 12 08:45:07 2018 -0800 Merge pull request #7242: Add a MonitoringInfoLabelProps proto to attach the proper key string name to MonitoringInfo label Enums .../fn-execution/src/main/proto/beam_fn_api.proto | 34 ++ 1 file changed, 29 insertions(+), 5 deletions(-)
[beam] branch master updated (5574f47 -> a61f2c5)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5574f47 Merge pull request #7243: [BEAM-6205] Setup gradle task ro run fnapi worker test with use_execu… add 8828b16 Add a MonitoringInfoLabelProps proto to attach the proper key string name to MonitoringInfo label Enums new a61f2c5 Merge pull request #7242: Add a MonitoringInfoLabelProps proto to attach the proper key string name to MonitoringInfo label Enums The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../fn-execution/src/main/proto/beam_fn_api.proto | 34 ++ 1 file changed, 29 insertions(+), 5 deletions(-)
[beam] branch master updated (798b3b3 -> 5574f47)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 798b3b3 Merge pull request #7241: [BEAM-6204] Clean up duplicated SocketAddressFactory class. add 5e506bf [BEAM-6205] Setup gradle task ro run fnapi worker test with use_executable_stage_bundle_execution new 5574f47 Merge pull request #7243: [BEAM-6205] Setup gradle task ro run fnapi worker test with use_execu… The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/google-cloud-dataflow-java/build.gradle | 4 +++- runners/google-cloud-dataflow-java/examples/build.gradle | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-)
[beam] 01/01: Merge pull request #7243: [BEAM-6205] Setup gradle task ro run fnapi worker test with use_execu…
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 5574f4739d2d205e909f8f6ad4bb6fdf8fa37857 Merge: 798b3b3 5e506bf Author: Scott Wegner AuthorDate: Wed Dec 12 08:35:47 2018 -0800 Merge pull request #7243: [BEAM-6205] Setup gradle task ro run fnapi worker test with use_execu… runners/google-cloud-dataflow-java/build.gradle | 4 +++- runners/google-cloud-dataflow-java/examples/build.gradle | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-)
[beam] branch master updated (a53f56a -> 798b3b3)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a53f56a Merge pull request #7261 from echauchot/BEAM-6216-flink-local add 9e8ac83 [BEAM-6240] Clean up duplicated SocketAddressFactory class. new 798b3b3 Merge pull request #7241: [BEAM-6204] Clean up duplicated SocketAddressFactory class. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../dataflow/worker/fn/SocketAddressFactory.java | 68 -- .../worker/fn/SocketAddressFactoryTest.java| 55 - 2 files changed, 123 deletions(-) delete mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/SocketAddressFactory.java delete mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/SocketAddressFactoryTest.java
[beam] 01/01: Merge pull request #7241: [BEAM-6204] Clean up duplicated SocketAddressFactory class.
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 798b3b3edb5549602d979f28afffea86649f0ed1 Merge: a53f56a 9e8ac83 Author: Scott Wegner AuthorDate: Wed Dec 12 08:34:20 2018 -0800 Merge pull request #7241: [BEAM-6204] Clean up duplicated SocketAddressFactory class. .../dataflow/worker/fn/SocketAddressFactory.java | 68 -- .../worker/fn/SocketAddressFactoryTest.java| 55 - 2 files changed, 123 deletions(-)
[beam] branch master updated: [BEAM-6178] Adding beam-sdks-java-bom, adding exportJavadoc flag for applyJavaNature (#7197)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new bfd1be9 [BEAM-6178] Adding beam-sdks-java-bom, adding exportJavadoc
flag for applyJavaNature (#7197)
bfd1be9 is described below
commit bfd1be9ae22d1ae7e732f590c448e9e5ed2894b9
Author: Garrett Jones
AuthorDate: Tue Dec 11 11:00:50 2018 -0800
[BEAM-6178] Adding beam-sdks-java-bom, adding exportJavadoc flag for
applyJavaNature (#7197)
---
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 89 ---
examples/java/build.gradle | 2 +-
runners/extensions-java/metrics/build.gradle | 2 +-
runners/flink/job-server/build.gradle | 1 +
.../examples-streaming/build.gradle| 2 +-
.../examples/build.gradle | 2 +-
.../google-cloud-dataflow-java/worker/build.gradle | 24 ++--
.../worker/legacy-worker/build.gradle | 1 +
runners/reference/job-server/build.gradle | 1 +
runners/samza/build.gradle | 2 +-
sdks/java/bom/build.gradle | 122 +
sdks/java/bom/pom.xml.template | 83 ++
sdks/java/build-tools/build.gradle | 2 +-
sdks/java/extensions/euphoria/build.gradle | 2 +-
sdks/java/extensions/kryo/build.gradle | 7 +-
sdks/java/extensions/sql/jdbc/build.gradle | 7 +-
sdks/java/io/common/build.gradle | 2 +-
sdks/java/io/file-based-io-tests/build.gradle | 2 +-
sdks/java/io/kudu/build.gradle | 2 +-
sdks/java/io/rabbitmq/build.gradle | 2 +-
sdks/java/io/synthetic/build.gradle| 2 +-
sdks/java/javadoc/build.gradle | 70
sdks/java/maven-archetypes/examples/build.gradle | 2 +-
sdks/java/maven-archetypes/starter/build.gradle| 2 +-
sdks/java/testing/load-tests/build.gradle | 2 +-
sdks/java/testing/nexmark/build.gradle | 2 +-
sdks/java/testing/test-utils/build.gradle | 2 +-
settings.gradle| 2 +
vendor/sdks-java-extensions-protobuf/build.gradle | 7 +-
29 files changed, 326 insertions(+), 122 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 329a6ed..acfffc9 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -123,6 +123,9 @@ class BeamModulePlugin implements Plugin {
/** Controls whether this project is published to Maven. */
boolean publish = true
+
+/** Controls whether javadoc is exported for this project. */
+boolean exportJavadoc = true
}
/** A class defining the set of configurable properties accepted by
applyPortabilityNature. */
@@ -489,6 +492,47 @@ class BeamModulePlugin implements Plugin {
relocate "com.google.thirdparty",
project.getJavaRelocatedPath("com.google.thirdparty")
}
+project.ext.repositories = {
+ maven {
+name "testPublicationLocal"
+url "file://${project.rootProject.projectDir}/testPublication/"
+ }
+ maven {
+url(project.properties['distMgmtSnapshotsUrl'] ?: isRelease(project)
+?
'https://repository.apache.org/service/local/staging/deploy/maven2'
+:
'https://repository.apache.org/content/repositories/snapshots')
+
+// We attempt to find and load credentials from ~/.m2/settings.xml
file that a user
+// has configured with the Apache release and snapshot staging
credentials.
+//
+//
+//
+// apache.releases.https
+// USER_TOKEN
+// PASS_TOKEN
+//
+//
+// apache.snapshots.https
+// USER_TOKEN
+// PASS_TOKEN
+//
+//
+//
+def settingsXml = new File(System.getProperty('user.home'),
'.m2/settings.xml')
+if (settingsXml.exists()) {
+ def serverId = (project.properties['distMgmtServerId'] ?:
isRelease(project)
+ ? 'apache.releases.https' : 'apache.snapshots.https')
+ def m2SettingCreds = new
XmlSlurper().parse(settingsXml).servers.server.find { server ->
serverId.equals(server.id.text()) }
+ if (m2SettingCreds) {
+credentials {
+ username m2SettingCreds.username.text()
+ password m2SettingCreds.
[beam] branch master updated (5850c00 -> bc859cc)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5850c00 Merge pull request #7186: [BEAM-5920] Add additional owners for Community Metrics add bc859cc [BEAM-6181] Reporting user counters via MonitoringInfos in Portable Dataflow Runner. (#7202) No new revisions were added by this update. Summary of changes: .../dataflow/worker/WorkItemStatusClient.java | 18 +- .../worker/fn/control/BeamFnMapTaskExecutor.java | 259 - .../control/RegisterAndProcessBundleOperation.java | 22 +- .../fn/control/BeamFnMapTaskExecutorTest.java | 203 +++- .../RegisterAndProcessBundleOperationTest.java | 60 - .../SingularProcessBundleProgressTrackerTest.java | 6 +- 6 files changed, 501 insertions(+), 67 deletions(-)
[beam] 01/01: Merge pull request #7186: [BEAM-5920] Add additional owners for Community Metrics
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 5850c00bb9a8e1c8d025f05567628ab3b32b0dbe Merge: ce15b25 5e94da3 Author: Scott Wegner AuthorDate: Thu Dec 6 11:24:34 2018 -0800 Merge pull request #7186: [BEAM-5920] Add additional owners for Community Metrics .test-infra/metrics/OWNERS | 3 +++ 1 file changed, 3 insertions(+)
[beam] branch master updated (ce15b25 -> 5850c00)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ce15b25 Merge pull request #7214: Update PortableTimersExecutionTest to use PAssert add 5e94da3 [BEAM-5920] Add additional owners for Community Metrics new 5850c00 Merge pull request #7186: [BEAM-5920] Add additional owners for Community Metrics The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .test-infra/metrics/OWNERS | 3 +++ 1 file changed, 3 insertions(+)
[beam] branch master updated (95d0ac5 -> a5b36c5)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 95d0ac5 Merge pull request #6930: [BEAM-5462] get rid of .options deprecation warnings in tests add a3d2611 [BEAM-2400] Use getJobId() consistently add a5b36c5 Merge pull request #7199: [BEAM-2400] Use getJobId() consistently No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[beam] branch master updated: [BEAM-6100] Collect metrics properly in Load tests (#7087)
This is an automated email from the ASF dual-hosted git repository.
scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 68be16a [BEAM-6100] Collect metrics properly in Load tests (#7087)
68be16a is described below
commit 68be16a34823b7d322f3cc0c6d3a3aa53bdecccb
Author: Łukasz Gajowy
AuthorDate: Wed Nov 28 18:07:57 2018 +0100
[BEAM-6100] Collect metrics properly in Load tests (#7087)
---
.../beam/sdk/loadtests/CoGroupByKeyLoadTest.java | 20 ++---
.../apache/beam/sdk/loadtests/CombineLoadTest.java | 22 +++--
.../beam/sdk/loadtests/GroupByKeyLoadTest.java | 13 ++-
.../org/apache/beam/sdk/loadtests/LoadTest.java| 16 ++--
.../apache/beam/sdk/loadtests/ParDoLoadTest.java | 9 +-
.../{MetricsMonitor.java => ByteMonitor.java} | 19 +++--
.../sdk/loadtests/metrics/MetricsPublisher.java| 12 ++-
.../{MetricsMonitor.java => TimeMonitor.java} | 19 +++--
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 27 +++---
.../beam/sdk/testutils/metrics/MetricsReader.java | 97 --
.../sdk/testutils/metrics/MetricsReaderTest.java | 52
11 files changed, 181 insertions(+), 125 deletions(-)
diff --git
a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CoGroupByKeyLoadTest.java
b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CoGroupByKeyLoadTest.java
index 85d27ea..0cb8f0e 100644
---
a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CoGroupByKeyLoadTest.java
+++
b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CoGroupByKeyLoadTest.java
@@ -22,7 +22,7 @@ import java.util.Optional;
import org.apache.beam.sdk.io.synthetic.SyntheticBoundedIO;
import
org.apache.beam.sdk.io.synthetic.SyntheticBoundedIO.SyntheticSourceOptions;
import org.apache.beam.sdk.io.synthetic.SyntheticStep;
-import org.apache.beam.sdk.loadtests.metrics.MetricsMonitor;
+import org.apache.beam.sdk.loadtests.metrics.ByteMonitor;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.Description;
import org.apache.beam.sdk.options.Validation;
@@ -92,22 +92,22 @@ public class CoGroupByKeyLoadTest extends
LoadTest
Optional syntheticStep =
createStep(options.getStepOptions());
PCollection> input =
-applyStepIfPresent(
-pipeline.apply("Read input",
SyntheticBoundedIO.readFrom(sourceOptions)),
-"Synthetic step for input",
-syntheticStep);
+pipeline.apply("Read input",
SyntheticBoundedIO.readFrom(sourceOptions));
+input = input.apply("Collect start time metrics (input)",
ParDo.of(runtimeMonitor));
+applyStepIfPresent(input, "Synthetic step for input", syntheticStep);
PCollection> coInput =
-applyStepIfPresent(
-pipeline.apply("Read co-input",
SyntheticBoundedIO.readFrom(coSourceOptions)),
-"Synthetic step for co-input",
-syntheticStep);
+pipeline.apply("Read co-input",
SyntheticBoundedIO.readFrom(coSourceOptions));
+coInput = coInput.apply("Collect start time metrics (co-input)",
ParDo.of(runtimeMonitor));
+applyStepIfPresent(coInput, "Synthetic step for co-input", syntheticStep);
KeyedPCollectionTuple.of(INPUT_TAG, input)
.and(CO_INPUT_TAG, coInput)
.apply("CoGroupByKey", CoGroupByKey.create())
.apply("Ungroup and reiterate", ParDo.of(new
UngroupAndReiterate(options.getIterations(
-.apply("Collect metrics", ParDo.of(new
MetricsMonitor(METRICS_NAMESPACE)));
+.apply(
+"Collect total bytes", ParDo.of(new ByteMonitor(METRICS_NAMESPACE,
"totalBytes.count")))
+.apply("Collect end time metrics", ParDo.of(runtimeMonitor));
}
private static class UngroupAndReiterate
diff --git
a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CombineLoadTest.java
b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CombineLoadTest.java
index a85f23b..316d626 100644
---
a/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CombineLoadTest.java
+++
b/sdks/java/testing/load-tests/src/main/java/org/apache/beam/sdk/loadtests/CombineLoadTest.java
@@ -25,7 +25,8 @@ import java.math.BigInteger;
import java.util.Optional;
import org.apache.beam.sdk.io.synthetic.SyntheticBoundedIO;
import org.apache.beam.sdk.io.synthetic.SyntheticStep;
-import org.apache.beam.sdk.loadtests.metrics.MetricsMonitor;
+import org.apache.beam.sdk.loadtests.metrics.ByteMonitor;
+import org.apache.beam.sdk.loadtests.metrics.TimeMonitor;
import org.apache.beam.sdk.options.Default;
import org.apache.beam.sdk.options.D
[beam] 01/01: Merge pull request #7073: Don't redundantly execute pre-commits as part of Jenkins post-commits
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 3cd4d8f51d017bb0b0793e42ba7ffa39d325bfa4 Merge: f82d940 8f6502e Author: Scott Wegner AuthorDate: Wed Nov 21 21:25:23 2018 -0800 Merge pull request #7073: Don't redundantly execute pre-commits as part of Jenkins post-commits .test-infra/jenkins/job_PostCommit_Python_Verify.groovy | 6 -- build.gradle| 2 -- sdks/python/build.gradle| 1 - 3 files changed, 9 deletions(-)
[beam] branch master updated (f82d940 -> 3cd4d8f)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f82d940 [BEAM-6058] Use vendored gRPC 1.13.1 dependency. (#7105) add 8f6502e Don't redundantly execute pre-commits as part of Jenkins post-commits new 3cd4d8f Merge pull request #7073: Don't redundantly execute pre-commits as part of Jenkins post-commits The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .test-infra/jenkins/job_PostCommit_Python_Verify.groovy | 6 -- build.gradle| 2 -- sdks/python/build.gradle| 1 - 3 files changed, 9 deletions(-)
[beam] 01/01: Merge pull request #7074: [BEAM-6084] Fix Dataflow runner task dependencies for container registration.
This is an automated email from the ASF dual-hosted git repository. scott pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit bf44ab1bc0c83e8c9829661bbcabb84d6820d577 Merge: 980735e c675500 Author: Scott Wegner AuthorDate: Tue Nov 20 13:10:44 2018 -0800 Merge pull request #7074: [BEAM-6084] Fix Dataflow runner task dependencies for container registration. runners/google-cloud-dataflow-java/build.gradle | 15 ++- 1 file changed, 6 insertions(+), 9 deletions(-)
[beam] branch master updated (980735e -> bf44ab1)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 980735e Move implementation protos to bottom of the file. (#7089) add c675500 [BEAM-6084] Fix Dataflow runner task dependencies for container registration. new bf44ab1 Merge pull request #7074: [BEAM-6084] Fix Dataflow runner task dependencies for container registration. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/google-cloud-dataflow-java/build.gradle | 15 ++- 1 file changed, 6 insertions(+), 9 deletions(-)
[beam] branch master updated (841f9eb -> 8f7f765)
This is an automated email from the ASF dual-hosted git repository. scott pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 841f9eb Merge pull request #7047: [BEAM-6049] Add option to load job to GCS in Dataflow Runner add bb7951d [BEAM-6073] Update README with lockfile & gradle instructions add 8f7f765 Merge pull request #7070: [BEAM-6073] Update README with lockfile & gradle instructions No new revisions were added by this update. Summary of changes: sdks/go/README.md | 20 1 file changed, 20 insertions(+)
