Build failed in Jenkins: beam_PostCommit_RunnableOnService_GearpumpLocal #153

[Apache Jenkins Server]

See 


Changes:

[thw] BEAM-261 Apex runner PoC

[thw] BEAM-261 Use Apex 3.5.0-SNAPSHOT to use loopback as connect address.

[thw] BEAM-261 Read.Bounded and FlattenPCollection.

[thw] BEAM-261 PCollectionView and side inputs.

[thw] BEAM-261 Add support for ParDo.BoundMulti

[thw] BEAM-261 Checkpointing for pushed back inputs.

[thw] BEAM-261 Support multiple side inputs.

[thw] BEAM-261 Enable checkstyle and cleanup.

[thw] BEAM-783 Add test to cover side inputs and outputs.

[thw] Adjust for merge from master.

[thw] BEAM-784 Checkpointing for StateInternals

[iemejia] [BEAM-815] Fix ApexPipelineOptions conversion and add

[thw] BEAM-261 Skip integration tests unless running with

[thw] Adjust for merge from master.

[thw] BEAM-858 Enable ApexRunner integration test in examples.

[thw] BEAM-844 Add README, wordcount test, fix View.AsIterable, initial GBK

[thw] Adjust for merge from master.

[thw] BEAM-261 Make translators package private.

[klk] Add OnTimerInvoker(s), for invoking DoFn @OnTimer methods

[thw] Fix findbugs issues.

[lcwik] Changes in AvroCoder serialization so it can serialize in Kryo

[lcwik] Swap to use Serializable ThreadLocal and serializable Schema holder.

[tgroh] Update Watermarks even if a Reader is empty

[tgroh] Add an Enforcement enum

[klk] Add Spark and Flink runner version to root pom

[klk] Add runner-specific profiles to the examples pom.xml

[klk] Hardcode MinimalWordCount to the DirectRunner

[klk] Revise WordCount example to be better cross-runner example

[klk] Revise DebuggingWordCount to be more portable

[diominay] fix exception during formatting

[klk] Fix findbugs errors and re-enable for runners-core

[aljoscha.krettek] [BEAM-931] Fix Findbugs Warnings in Flink Runner

[aljoscha.krettek] Remove Leftover Sysout Printing in Flink Runner

[dhalperi] Fix DatastoreIT test options

[tgroh] Use Credentials from GcpOptions instead of BigtableOptions

[klk] Fix breakage in WindowedWordCountIT

[klk] Fix null or empty check in TestDataflowRunner

[klk] Make TestDataflowRunner crash message actionable

[klk] Make test timeout configurable and use in TestDataflowRunner

[klk] Increase WindowedWordCountIT timeout to 20 minutes

[dhalperi] [BEAM-896] FileBasedSink removes temp directory

[tgroh] Update BigtableWriteIT to set Credentials

[ansela] [BEAM-944] Spark runner causes an exception when creating pipeline

[klk] Add script to update examples archetype automatically

[klk] Update examples archetype to match examples

[klk] fixup! Add script

[davor] [BEAM-962] Fix games example pipeline options default values conflicts.

[klk] Remove DoFnSignatures.INSTANCE

[klk] Remove OnTimerInvokers.INSTANCE; deprecate DoFnInvokers.INSTANCE

[davor] [BEAM-856] Use free network port for the Derby test instance

[tgroh] Use a Multiset to track Pending Bundles

[tgroh] Reduce Verbosity of testFlattenInputMultipleCopies

[thw] Add beam-runners-apex dependency.

[tgroh] Do not pass Credentials in BigtableWriteIT DoFn

[davor] [BEAM-551] Add better toString methods for VPs

[dhalperi] Make BigQueryIO.parseTableSpec public

[dhalperi] [BEAM-954] FileBasedSink: remove unused code of 
TemporaryFileRetention.

[davor] WordCountIT: uses input with fewer keys.

[dhalperi] Use Avro serializer for Kafka checkpoint mark.

[tgroh] Fix FindBugs Errors in the Direct Runner

[tgroh] Actually Split Root Transforms

[davor] [BEAM-856] Use free available port to start the test MongoDB instance

[aljoscha.krettek] [BEAM-965] Set Correct Output Type on Sources in Flink 
Stream Runner

[tgroh] Simplify FileOperations interface

[klk] Treat ProcessContext and Context like other DoFn parameters

[jbonofre] [BEAM-948] Add ability to write files to GridFS

[ansela] [BEAM-762] Unify spark-runner EvaluationContext and

[dhalperi] [BEAM-930] Fix findbugs and re-enable Maven plugin in MongoDbIO

[dhalperi] Revert "Closes #1356"

[jbonofre] [BEAM-930] Fix findbugs and re-enable Maven plugin in MongoDbIO and

[ansela] [BEAM-891] fix build occasionally fails on IndexOutOfBoundsException.

[klk] Rename DoFn.ExtraContextFactory to DoFn.ArgumentProvider

[ansela] [BEAM-983] Fix a bunch of precommit errors from #1332

[klk] Connect generated DoFnInvoker.invokerOnTimer to OnTimerInvoker

[tgroh] Exercise Dynamic Splitting in the DirectRunner

[tgroh] Remove Pipeline#getFullNameForTesting

[tgroh] Reduce incidence of Namespace StringKey comparisons

[tgroh] Remove unused Thread variable in TransformExecutor

[tgroh] Reduce Visibility of PCollectionViewWriter

[kirpichov] Adds used but undeclared dependencies to archetype

[tgroh] Properly apply Transform Overrides in the Direct Runner

[davor] [BEAM-895] Allow empty GCP credential for pipelines that access to

[davor] [BEAM-951] FileBasedSink: merge FileOperations into IOChannelFactory.

[davor] [BEAM-951] Inline copyOne() and improve comments.

[kirpichov] Liberates ReduceFnRunner fr

[20/50] incubator-beam git commit: [Beam-1001] Add non blocking cancel() and waitUntilFinish() for streaming applications. remove timeout parameer in spark pipeline option.

[kenn]

[Beam-1001] Add non blocking cancel() and waitUntilFinish() for streaming 
applications.
remove timeout parameer in spark pipeline option.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/dafd5be7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/dafd5be7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/dafd5be7

Branch: refs/heads/gearpump-runner
Commit: dafd5be7f69f191fc9edb8b9f9aec010ca368f50
Parents: d93e9a8
Author: ksalant 
Authored: Sun Nov 20 11:57:16 2016 +0200
Committer: Sela 
Committed: Sun Nov 20 19:25:52 2016 +0200

--
 .../runners/spark/SparkPipelineOptions.java |  6 --
 .../apache/beam/runners/spark/SparkRunner.java  |  5 +-
 .../spark/translation/EvaluationContext.java| 59 +---
 .../SparkRunnerStreamingContextFactory.java |  3 +-
 .../streaming/EmptyStreamAssertionTest.java |  3 +-
 .../streaming/FlattenStreamingTest.java |  4 +-
 .../streaming/KafkaStreamingTest.java   | 11 ++--
 .../ResumeFromCheckpointStreamingTest.java  |  3 +-
 .../streaming/SimpleStreamingWordCountTest.java |  4 +-
 .../streaming/utils/PAssertStreaming.java   |  8 ++-
 .../SparkTestPipelineOptionsForStreaming.java   |  1 -
 11 files changed, 60 insertions(+), 47 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/dafd5be7/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
index b1ebde9..0fd790e 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
@@ -39,12 +39,6 @@ public interface SparkPipelineOptions
   String getSparkMaster();
   void setSparkMaster(String master);
 
-  @Description("Timeout to wait (in msec) for a streaming execution to stop, 
-1 runs until "
-  + "execution is stopped")
-  @Default.Long(-1)
-  Long getTimeout();
-  void setTimeout(Long timeoutMillis);
-
   @Description("Batch interval for Spark streaming in milliseconds.")
   @Default.Long(1000)
   Long getBatchIntervalMillis();

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/dafd5be7/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
index 6bbef39..e800071 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunner.java
@@ -137,11 +137,8 @@ public final class SparkRunner extends 
PipelineRunner {
 
 // if recovering from checkpoint, we have to reconstruct the 
EvaluationResult instance.
 return contextFactory.getCtxt() == null ? new 
EvaluationContext(jssc.sc(),
-pipeline, jssc, mOptions.getTimeout()) : contextFactory.getCtxt();
+pipeline, jssc) : contextFactory.getCtxt();
   } else {
-if (mOptions.getTimeout() > 0) {
-  LOG.info("Timeout is ignored by the SparkRunner in batch.");
-}
 JavaSparkContext jsc = SparkContextFactory.getSparkContext(mOptions);
 EvaluationContext ctxt = new EvaluationContext(jsc, pipeline);
 SparkPipelineTranslator translator = new 
TransformTranslator.Translator();

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/dafd5be7/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
index aaf7573..1183fbb 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
@@ -45,6 +45,7 @@ import org.apache.beam.sdk.values.POutput;
 import org.apache.beam.sdk.values.PValue;
 import org.apache.spark.api.java.JavaRDD;
 import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.streaming.StreamingContextState;
 import org.apache.spark.streaming.api.java.JavaStreamingContext;
 import org.joda.time.Duration;
 
@@ -57,7 +58,6 @@ public class Ev

[23/50] incubator-beam git commit: [BEAM-1018] This closes #1394

[kenn]

[BEAM-1018] This closes #1394


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c6b86f60
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c6b86f60
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c6b86f60

Branch: refs/heads/gearpump-runner
Commit: c6b86f60f7b159fd188bfb6e5bf742a9b65080a0
Parents: 875631f a761b0c
Author: Jean-Baptiste Onofré 
Authored: Mon Nov 21 20:52:10 2016 +0100
Committer: Jean-Baptiste Onofré 
Committed: Mon Nov 21 20:52:10 2016 +0100

--
 .../src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[32/50] incubator-beam git commit: Add JUnit category for stateful ParDo tests

[kenn]

Add JUnit category for stateful ParDo tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8d715689
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8d715689
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8d715689

Branch: refs/heads/gearpump-runner
Commit: 8d715689dd5283b7b180c0b9ec4e188abba140f5
Parents: 70efa47
Author: Kenneth Knowles 
Authored: Mon Nov 21 15:41:13 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 21 21:34:19 2016 -0800

--
 .../beam/sdk/testing/UsesStatefulParDo.java | 25 
 1 file changed, 25 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8d715689/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStatefulParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStatefulParDo.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStatefulParDo.java
new file mode 100644
index 000..8bd6330
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStatefulParDo.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+import org.apache.beam.sdk.transforms.ParDo;
+
+/**
+ * Category tag for validation tests which utilize stateful {@link ParDo}.
+ */
+public interface UsesStatefulParDo {}

[47/50] incubator-beam git commit: Simplify the API for managing MetricsEnvironment

[kenn]

Simplify the API for managing MetricsEnvironment

1. setCurrentContainer returns the previous MetricsEnvironment
2. setCurrentContainer(null) resets the thread local
3. scopedCurrentContainer sets the container and returns a Closeable to
   reset the previous container.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e6870a6d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e6870a6d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e6870a6d

Branch: refs/heads/gearpump-runner
Commit: e6870a6dc10e4ad52a911c316137a9f7731a9194
Parents: 6ec45f7
Author: bchambers 
Authored: Tue Nov 22 11:37:23 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 22 12:27:41 2016 -0800

--
 .../beam/runners/direct/TransformExecutor.java  |  5 +-
 .../beam/sdk/metrics/MetricsEnvironment.java| 60 +++-
 .../sdk/metrics/MetricsEnvironmentTest.java |  8 +--
 .../apache/beam/sdk/metrics/MetricsTest.java|  6 +-
 4 files changed, 56 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e6870a6d/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
index 1704955..fb31cc9 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.direct;
 
+import java.io.Closeable;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.concurrent.Callable;
@@ -89,8 +90,7 @@ class TransformExecutor implements Runnable {
   @Override
   public void run() {
 MetricsContainer metricsContainer = new 
MetricsContainer(transform.getFullName());
-MetricsEnvironment.setMetricsContainer(metricsContainer);
-try {
+try (Closeable metricsScope = 
MetricsEnvironment.scopedMetricsContainer(metricsContainer)) {
   Collection> enforcements = new ArrayList<>();
   for (ModelEnforcementFactory enforcementFactory : modelEnforcements) {
 ModelEnforcement enforcement = 
enforcementFactory.forBundle(inputBundle, transform);
@@ -117,7 +117,6 @@ class TransformExecutor implements Runnable {
   // Report the physical metrics from the end of this step.
   context.getMetrics().commitPhysical(inputBundle, 
metricsContainer.getCumulative());
 
-  MetricsEnvironment.unsetMetricsContainer();
   transformEvaluationState.complete(this);
 }
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e6870a6d/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
index ef2660a8..7c06cbf 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
@@ -17,6 +17,8 @@
  */
 package org.apache.beam.sdk.metrics;
 
+import java.io.Closeable;
+import java.io.IOException;
 import java.util.concurrent.atomic.AtomicBoolean;
 import javax.annotation.Nullable;
 import org.slf4j.Logger;
@@ -29,11 +31,13 @@ import org.slf4j.LoggerFactory;
  * returned objects to create and modify metrics.
  *
  * The runner should create {@link MetricsContainer} for each context in 
which metrics are
- * reported (by step and name) and call {@link #setMetricsContainer} before 
invoking any code that
- * may update metrics within that step.
+ * reported (by step and name) and call {@link #setCurrentContainer} before 
invoking any code that
+ * may update metrics within that step. It should call {@link 
#setCurrentContainer} again to restore
+ * the previous container.
  *
- * The runner should call {@link #unsetMetricsContainer} (or {@link 
#setMetricsContainer} back to
- * the previous value) when exiting code that set the metrics container.
+ * Alternatively, the runner can use {@link 
#scopedMetricsContainer(MetricsContainer)} to set the
+ * container for the current thread and get a {@link Closeable} that will 
restore the previous
+ * container when closed.
  */
 public class MetricsEnvironment {
 
@@ -45,15 +49,20 @@ public class MetricsEnvironment {
   private static final ThreadLocal CONTAINER_FOR_THREAD =
   new ThreadLocal();
 
-  /**

[28/50] incubator-beam git commit: Remove microbenchmarks directory from Beam

[kenn]

Remove microbenchmarks directory from Beam

The microbenchmarks should not be released as part of the Beam
distribution, rather they exist for internal measurements and testing.

Given that we also don't have any automatic monitoring (and that any
such measurements are not indicative of real performance), the value of
maintaining microbenchmarks here is negligible.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1b520708
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1b520708
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1b520708

Branch: refs/heads/gearpump-runner
Commit: 1b5207084942e3eeefd1f35c5e34f7943704e19e
Parents: 869d082
Author: bchambers 
Authored: Mon Nov 21 13:35:39 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 14:32:27 2016 -0800

--
 sdks/java/microbenchmarks/README.md |  42 
 sdks/java/microbenchmarks/pom.xml   | 113 -
 .../coders/AvroCoderBenchmark.java  | 119 --
 .../coders/ByteArrayCoderBenchmark.java |  64 -
 .../coders/CoderBenchmarking.java   |  41 
 .../coders/StringUtf8CoderBenchmark.java|  70 --
 .../microbenchmarks/coders/package-info.java|  22 --
 .../transforms/DoFnInvokersBenchmark.java   | 232 ---
 .../transforms/package-info.java|  22 --
 sdks/java/pom.xml   |   1 -
 10 files changed, 726 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b520708/sdks/java/microbenchmarks/README.md
--
diff --git a/sdks/java/microbenchmarks/README.md 
b/sdks/java/microbenchmarks/README.md
deleted file mode 100644
index 627e669..000
--- a/sdks/java/microbenchmarks/README.md
+++ /dev/null
@@ -1,42 +0,0 @@
-
-
-# Microbenchmarks for parts of the Beam SDK
-
-To run benchmarks:
-
- 1. Run `mvn install` in the top directory to install the SDK.
-
- 2. Build the benchmark package:
-
-cd microbenchmarks
-mvn package
-
- 3. run benchmark harness:
-
-java -jar target/microbenchmarks.jar
-
- 4. (alternate to step 3)
-to run just a subset of benchmarks, pass a regular expression that
-matches the benchmarks you want to run (this can match against the class
-name, or the method name).  E.g., to run any benchmarks with
-"DoFnReflector" in the name:
-
-java -jar target/microbenchmarks.jar ".*DoFnReflector.*"
-

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b520708/sdks/java/microbenchmarks/pom.xml
--
diff --git a/sdks/java/microbenchmarks/pom.xml 
b/sdks/java/microbenchmarks/pom.xml
deleted file mode 100644
index 06bc4df..000
--- a/sdks/java/microbenchmarks/pom.xml
+++ /dev/null
@@ -1,113 +0,0 @@
-
-
-http://maven.apache.org/POM/4.0.0"; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
-
-  4.0.0
-
-  
-org.apache.beam
-beam-sdks-java-parent
-0.4.0-incubating-SNAPSHOT
-../pom.xml
-  
-
-  beam-sdks-java-microbenchmarks
-  Apache Beam :: SDKs :: Java :: Microbenchmarks
-  Microbenchmarks for components in the Beam Java 
SDK.
-  jar
-
-  
-
-  
-
-
-  org.codehaus.mojo
-  findbugs-maven-plugin
-  
-true
-  
-
-  
-
-
-
-  
-org.apache.maven.plugins
-maven-compiler-plugin
-  
-
-  
-org.apache.maven.plugins
-maven-jar-plugin
-  
-
-  
-org.apache.maven.plugins
-maven-shade-plugin
-
-  
-package
-
-  shade
-
-
-  microbenchmarks
-  
-
-  org.openjdk.jmh.Main
-
-  
-
-  
-
-  
-
-  
-
-  
-
-  org.apache.beam
-  beam-sdks-java-core
-
-
-
-  joda-time
-  joda-time
-
-
-
-  org.slf4j
-  slf4j-jdk14
-  
-  runtime
-
-
-
-  org.openjdk.jmh
-  jmh-core
-  1.0.1
-
-
-
-  org.openjdk.jmh
-  jmh-generator-annprocess
-  1.6.1
-  provided
-
-  
-

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b520708/sdks/java/microbenchmarks/src/main/java/org/apache/beam/sdk/microbenchmarks/coders/AvroCoderBenchmark.java
--
diff --git 
a/sdks/java/microbenchmarks/src/main/java/org/apache/beam/sdk/microbenchmarks/coders/AvroCoderBenchm

[04/50] incubator-beam git commit: Block earlier in BoundedReadEvaluatorFactoryTest

[kenn]

Block earlier in BoundedReadEvaluatorFactoryTest

This ensures that the reader doesn't claim the split point, which in
turn ensures the dynamic split request will not be refused by the
OffsetBasedSource. If the split is refused, ...ProducesDynamicSplits
flakes, as if the reader is faster than the split thread it can run past
the point at which the splitter thread will attempt to split the source,
which causes the reader to read all of the elements.

Sleep within TestReader#advanceImpl if the reader is being dynamically
split, to ensure that the dynamic split fully completes before
continuing a call to advance.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4a8d32e5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4a8d32e5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4a8d32e5

Branch: refs/heads/gearpump-runner
Commit: 4a8d32e5d3726b851329d507a8d0392cc03f6e85
Parents: 1543ea9
Author: Thomas Groh 
Authored: Thu Nov 17 10:56:49 2016 -0800
Committer: Thomas Groh 
Committed: Thu Nov 17 14:37:47 2016 -0800

--
 .../direct/BoundedReadEvaluatorFactoryTest.java | 26 +++-
 1 file changed, 14 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4a8d32e5/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
index 9d8503a..e956c34 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactoryTest.java
@@ -49,6 +49,7 @@ import org.apache.beam.sdk.io.CountingSource;
 import org.apache.beam.sdk.io.OffsetBasedSource;
 import org.apache.beam.sdk.io.OffsetBasedSource.OffsetBasedReader;
 import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.Source;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.testing.SourceTestUtils;
@@ -142,9 +143,7 @@ public class BoundedReadEvaluatorFactoryTest {
 TestPipeline.create().apply(Read.from(new 
TestSource<>(VarLongCoder.of(), 5, elems)));
 AppliedPTransform transform = 
read.getProducingTransformInternal();
 Collection> unreadInputs =
-new BoundedReadEvaluatorFactory.InputProvider(context)
-.getInitialInputs(transform,
-1);
+new 
BoundedReadEvaluatorFactory.InputProvider(context).getInitialInputs(transform, 
1);
 
 Collection> outputs = new ArrayList<>();
 int numIterations = 0;
@@ -155,8 +154,7 @@ public class BoundedReadEvaluatorFactoryTest {
 
   Collection> newUnreadInputs = new ArrayList<>();
   for (CommittedBundle shardBundle : unreadInputs) {
-TransformEvaluator evaluator =
-factory.forApplication(transform, null);
+TransformEvaluator evaluator = factory.forApplication(transform, 
null);
 for (WindowedValue shard : shardBundle.getElements()) {
   evaluator.processElement((WindowedValue) shard);
 }
@@ -178,8 +176,6 @@ public class BoundedReadEvaluatorFactoryTest {
   unreadInputs = newUnreadInputs;
 }
 
-// We produced at least one split before we read 1000 elements, as we will 
attempt to split as
-// quickly as possible.
 assertThat(numIterations, greaterThan(1));
 WindowedValue[] expectedValues = new WindowedValue[numElements];
 for (long i = 0L; i < numElements; i++) {
@@ -343,7 +339,7 @@ public class BoundedReadEvaluatorFactoryTest {
 private static boolean readerClosed;
 private final Coder coder;
 private final T[] elems;
-private final int awaitSplitIndex;
+private final int firstSplitIndex;
 
 private transient CountDownLatch subrangesCompleted;
 
@@ -351,11 +347,11 @@ public class BoundedReadEvaluatorFactoryTest {
   this(coder, elems.length, elems);
 }
 
-public TestSource(Coder coder, int awaitSplitIndex, T... elems) {
+public TestSource(Coder coder, int firstSplitIndex, T... elems) {
   super(0L, elems.length, 1L);
   this.elems = elems;
   this.coder = coder;
-  this.awaitSplitIndex = awaitSplitIndex;
+  this.firstSplitIndex = firstSplitIndex;
   readerClosed = false;
 
   subrangesCompleted = new CountDownLatch(2);
@@ -380,7 +376,7 @@ public class BoundedReadEvaluatorFactoryTest {
 @Override
 public BoundedSource.BoundedReader createReade

[27/50] incubator-beam git commit: This closes #1401

[kenn]

This closes #1401


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/869d082e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/869d082e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/869d082e

Branch: refs/heads/gearpump-runner
Commit: 869d082e9c1e0807ef282f0ac617c140cd2a3181
Parents: cb5d216 d39471b
Author: Davor Bonaci 
Authored: Mon Nov 21 14:30:13 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 14:30:13 2016 -0800

--
 pom.xml |  34 +---
 runners/apex/pom.xml|   2 +-
 runners/core-java/pom.xml   |   2 +-
 .../beam/runners/core/PaneInfoTracker.java  |   3 -
 .../beam/runners/core/SystemReduceFn.java   |   3 -
 .../apache/beam/runners/core/WatermarkHold.java |   3 -
 .../AfterDelayFromFirstElementStateMachine.java |   5 -
 .../core/triggers/AfterPaneStateMachine.java|   3 -
 .../core/triggers/TriggerStateMachines.java |  15 --
 runners/direct-java/pom.xml |   2 +-
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../src/main/resources/beam/findbugs-filter.xml | 172 ++-
 sdks/java/core/pom.xml  |   2 +-
 .../org/apache/beam/sdk/coders/AvroCoder.java   |   5 -
 .../apache/beam/sdk/coders/InstantCoder.java|   4 -
 .../org/apache/beam/sdk/testing/PAssert.java|   3 -
 .../beam/sdk/testing/SerializableMatchers.java  |   3 -
 ...AttemptAndTimeBoundedExponentialBackOff.java |   3 -
 .../sdk/util/ExposedByteArrayInputStream.java   |   2 -
 .../sdk/util/ExposedByteArrayOutputStream.java  |   3 -
 .../java/org/apache/beam/sdk/util/ZipFiles.java |   9 -
 .../beam/sdk/util/state/StateMerging.java   |   3 -
 sdks/java/io/google-cloud-platform/pom.xml  |  14 +-
 sdks/java/io/hdfs/pom.xml   |   2 +-
 sdks/java/io/jdbc/pom.xml   |   2 +-
 sdks/java/io/jms/pom.xml|   2 +-
 sdks/java/io/kafka/pom.xml  |   2 +-
 sdks/java/io/kinesis/pom.xml|   2 +-
 sdks/java/io/mongodb/pom.xml|   2 +-
 29 files changed, 178 insertions(+), 131 deletions(-)
--

[09/50] incubator-beam git commit: Remove dead code from MetricsContainer

[kenn]

Remove dead code from MetricsContainer


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5fc68a5c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5fc68a5c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5fc68a5c

Branch: refs/heads/gearpump-runner
Commit: 5fc68a5c109fdbe3ed8afb370c8fcee803e9f1d8
Parents: 4852d2e
Author: bchambers 
Authored: Thu Nov 17 14:40:52 2016 -0800
Committer: bchambers 
Committed: Fri Nov 18 09:55:29 2016 -0800

--
 .../org/apache/beam/sdk/metrics/MetricsContainer.java  | 13 -
 1 file changed, 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5fc68a5c/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsContainer.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsContainer.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsContainer.java
index 10032a2..ba5a343 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsContainer.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsContainer.java
@@ -130,19 +130,6 @@ public class MetricsContainer {
* container.
*/
   public MetricUpdates getCumulative() {
-ImmutableList.Builder> counterUpdates = 
ImmutableList.builder();
-for (Map.Entry counter : counters.entries()) {
-  counterUpdates.add(MetricUpdate.create(
-  MetricKey.create(stepName, counter.getKey()), 
counter.getValue().getCumulative()));
-}
-
-ImmutableList.Builder> distributionUpdates =
-ImmutableList.builder();
-for (Map.Entry distribution : 
distributions.entries()) {
-  distributionUpdates.add(MetricUpdate.create(
-  MetricKey.create(stepName, distribution.getKey()),
-  distribution.getValue().getCumulative()));
-}
 return MetricUpdates.create(
 extractCumulatives(counters),
 extractCumulatives(distributions));

[45/50] incubator-beam git commit: Closes #1410

[kenn]

Closes #1410


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6ec45f7e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6ec45f7e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6ec45f7e

Branch: refs/heads/gearpump-runner
Commit: 6ec45f7e732ede0e2f8256df7a483a017461594d
Parents: e7d7aa9 e85cea7
Author: Thomas Weise 
Authored: Tue Nov 22 11:03:56 2016 -0800
Committer: Thomas Weise 
Committed: Tue Nov 22 11:03:56 2016 -0800

--
 runners/apex/pom.xml|  1 +
 .../translation/ParDoBoundMultiTranslator.java  | 67 +---
 .../apex/translation/ParDoBoundTranslator.java  | 46 +-
 3 files changed, 74 insertions(+), 40 deletions(-)
--

[10/50] incubator-beam git commit: Move DoFn.ArgumentProvider to DoFnInvoker.ArgumentProvider

[kenn]

Move DoFn.ArgumentProvider to DoFnInvoker.ArgumentProvider

The arguments provided as a single object are an aspect of the
DoFnInvoker, not the DoFn. The DoFn itself is a specification
that may have other ways of being invoked, depending on the
circumstance.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/33fb8c2d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/33fb8c2d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/33fb8c2d

Branch: refs/heads/gearpump-runner
Commit: 33fb8c2db8c64275f1b9b8ac6dfd12e92d7fb777
Parents: bb9c386
Author: Kenneth Knowles 
Authored: Thu Nov 17 23:04:55 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 14:20:20 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java |   4 +-
 .../beam/runners/core/SplittableParDo.java  |   7 +-
 .../org/apache/beam/sdk/transforms/DoFn.java| 122 ---
 .../beam/sdk/transforms/DoFnAdapters.java   |  10 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java|  41 +++
 .../sdk/transforms/reflect/DoFnInvoker.java | 121 +-
 .../sdk/transforms/reflect/DoFnInvokers.java|   4 +-
 .../sdk/transforms/reflect/OnTimerInvoker.java  |   8 +-
 .../transforms/reflect/DoFnInvokersTest.java|   5 +-
 .../transforms/reflect/OnTimerInvokersTest.java |   2 +-
 .../transforms/DoFnInvokersBenchmark.java   |   5 +-
 11 files changed, 161 insertions(+), 168 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33fb8c2d/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index 76aae8f..841e412 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -183,7 +183,7 @@ public class SimpleDoFnRunner implements 
DoFnRunner the type of the {@link DoFn} (main) output elements
*/
   private static class DoFnContext extends DoFn.Context
-  implements DoFn.ArgumentProvider {
+  implements DoFnInvoker.ArgumentProvider {
 private static final int MAX_SIDE_OUTPUTS = 1000;
 
 final PipelineOptions options;
@@ -424,7 +424,7 @@ public class SimpleDoFnRunner implements 
DoFnRunner the type of the {@link DoFn} (main) output elements
*/
   private class DoFnProcessContext extends DoFn.ProcessContext
-  implements DoFn.ArgumentProvider {
+  implements DoFnInvoker.ArgumentProvider {
 
 final DoFn fn;
 final DoFnContext context;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33fb8c2d/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
index 3003984..c38ab2f 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
@@ -392,10 +392,11 @@ public class SplittableParDo<
 }
 
 /**
- * Creates an {@link DoFn.ArgumentProvider} that provides the given 
tracker as well as the given
+ * Creates an {@link DoFnInvoker.ArgumentProvider} that provides the given 
tracker as well as
+ * the given
  * {@link ProcessContext} (which is also provided when a {@link Context} 
is requested.
  */
-private DoFn.ArgumentProvider wrapTracker(
+private DoFnInvoker.ArgumentProvider wrapTracker(
 TrackerT tracker, DoFn.ProcessContext processContext) 
{
 
   return new ArgumentProviderForTracker<>(tracker, processContext);
@@ -403,7 +404,7 @@ public class SplittableParDo<
 
 private static class ArgumentProviderForTracker<
 InputT, OutputT, TrackerT extends RestrictionTracker>
-implements DoFn.ArgumentProvider {
+implements DoFnInvoker.ArgumentProvider {
   private final TrackerT tracker;
   private final DoFn.ProcessContext processContext;
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33fb8c2d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
index bf0631b..997

[15/50] incubator-beam git commit: DirectRunner: Expand ParDo.Bound into ParDo.BoundMulti

[kenn]

DirectRunner: Expand ParDo.Bound into ParDo.BoundMulti


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f86e98c9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f86e98c9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f86e98c9

Branch: refs/heads/gearpump-runner
Commit: f86e98c91cedbb1d9fd54e3268dfd8f014ac2f27
Parents: 34e2a35
Author: Kenneth Knowles 
Authored: Wed Nov 16 15:43:47 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../beam/runners/direct/DirectRunner.java   |   3 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |  56 +--
 .../direct/ParDoMultiEvaluatorHooks.java|  55 ---
 .../direct/ParDoMultiOverrideFactory.java   |  51 +++
 .../runners/direct/ParDoOverrideFactory.java|  53 ---
 .../direct/ParDoSingleEvaluatorHooks.java   |  58 ---
 .../ParDoSingleViaMultiOverrideFactory.java |  66 +++
 .../direct/TransformEvaluatorRegistry.java  |   7 +-
 .../direct/ParDoMultiEvaluatorHooksTest.java| 439 ---
 .../direct/ParDoSingleEvaluatorHooksTest.java   | 335 --
 10 files changed, 139 insertions(+), 984 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f86e98c9/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index cce73c3..0060e84 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -87,7 +87,8 @@ public class DirectRunner
   .put(GroupByKey.class, new DirectGroupByKeyOverrideFactory())
   .put(TestStream.class, new DirectTestStreamFactory())
   .put(Write.Bound.class, new WriteWithShardingFactory())
-  .put(ParDo.Bound.class, new ParDoOverrideFactory())
+  .put(ParDo.Bound.class, new ParDoSingleViaMultiOverrideFactory())
+  .put(ParDo.BoundMulti.class, new ParDoMultiOverrideFactory())
   .put(
   GBKIntoKeyedWorkItems.class,
   new DirectGBKIntoKeyedWorkItemsOverrideFactory())

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f86e98c9/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
index ee4987f..f126000 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
@@ -24,49 +24,22 @@ import 
org.apache.beam.runners.direct.DirectExecutionContext.DirectStepContext;
 import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.ParDo.BoundMulti;
 import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.PCollectionTuple;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-/**
- * A {@link TransformEvaluatorFactory} for {@link ParDo}-like primitive {@link 
PTransform
- * PTransforms}, parameterized by some {@link TransformHooks 
transform-specific handling}.
- */
-final class ParDoEvaluatorFactory<
-InputT,
-OutputT,
-TransformOutputT extends POutput,
-TransformT extends PTransform, 
TransformOutputT>>
-implements TransformEvaluatorFactory {
-  interface TransformHooks<
-  InputT,
-  OutputT,
-  TransformOutputT extends POutput,
-  TransformT extends PTransform, 
TransformOutputT>> {
-/** Returns the {@link DoFn} contained in the given {@link ParDo} 
transform. */
-DoFn getDoFn(TransformT transform);
-
-/** Configures and creates a {@link ParDoEvaluator} for the given {@link 
DoFn}. */
-ParDoEvaluator createParDoEvaluator(
-EvaluationContext evaluationContext,
-AppliedPTransform, TransformOutputT, TransformT> 
application,
-DirectStepContext stepContext,
-DoFn fnLocal);
-  }
+/** A {@link TransformEvaluatorFactory

[GitHub] incubator-beam pull request #1413: merge master into gearpump-runner

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1413


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[40/50] incubator-beam git commit: Update StarterPipeline

[kenn]

Update StarterPipeline

Convert StarterPipeline ParDo to MapElements.

Use the new DoFn for non-outputting transforms.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c80554b8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c80554b8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c80554b8

Branch: refs/heads/gearpump-runner
Commit: c80554b83426a585c762143e0ad533a73c2c3f0f
Parents: e53d6d4
Author: Scott Wegner 
Authored: Mon Nov 21 16:33:07 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 22 10:09:12 2016 -0800

--
 .../src/main/java/StarterPipeline.java| 18 ++
 .../src/main/java/it/pkg/StarterPipeline.java | 18 ++
 2 files changed, 20 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c80554b8/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/src/main/java/StarterPipeline.java
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/src/main/java/StarterPipeline.java
 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/src/main/java/StarterPipeline.java
index 0b21aa6..d6afdec 100644
--- 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/src/main/java/StarterPipeline.java
+++ 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/src/main/java/StarterPipeline.java
@@ -20,13 +20,15 @@ package ${package};
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.OldDoFn;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.MapElements;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SimpleFunction;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
- * A starter example for writing Google Cloud Dataflow programs.
+ * A starter example for writing Beam programs.
  *
  * The example takes two strings, converts them to their upper-case
  * representation and logs them.
@@ -39,7 +41,7 @@ import org.slf4j.LoggerFactory;
  * Platform, you should specify the following command-line options:
  *   --project=
  *   --stagingLocation=
- *   --runner=BlockingDataflowRunner
+ *   --runner=DataflowRunner
  */
 public class StarterPipeline {
   private static final Logger LOG = 
LoggerFactory.getLogger(StarterPipeline.class);
@@ -49,14 +51,14 @@ public class StarterPipeline {
 PipelineOptionsFactory.fromArgs(args).withValidation().create());
 
 p.apply(Create.of("Hello", "World"))
-.apply(ParDo.of(new OldDoFn() {
+.apply(MapElements.via(new SimpleFunction() {
   @Override
-  public void processElement(ProcessContext c) {
-c.output(c.element().toUpperCase());
+  public String apply(String input) {
+return input.toUpperCase();
   }
 }))
-.apply(ParDo.of(new OldDoFn() {
-  @Override
+.apply(ParDo.of(new DoFn() {
+  @ProcessElement
   public void processElement(ProcessContext c)  {
 LOG.info(c.element());
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c80554b8/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/src/main/java/it/pkg/StarterPipeline.java
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/src/main/java/it/pkg/StarterPipeline.java
 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/src/main/java/it/pkg/StarterPipeline.java
index b332442..4ae92e8 100644
--- 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/src/main/java/it/pkg/StarterPipeline.java
+++ 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/src/main/java/it/pkg/StarterPipeline.java
@@ -20,13 +20,15 @@ package it.pkg;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.OldDoFn;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.MapElements;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SimpleFunction;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
- * A starter example for writing Google Cloud Dataflow programs.
+ * A starter example for writing Beam programs.
  *
  * The example takes two strings, converts them to their upper-case
  * representation an

[05/50] incubator-beam git commit: This closes #1377

[kenn]

This closes #1377


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/212fec4e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/212fec4e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/212fec4e

Branch: refs/heads/gearpump-runner
Commit: 212fec4eba053fe603238a13ca6e0d9cb6aef697
Parents: 1543ea9 4a8d32e
Author: Thomas Groh 
Authored: Thu Nov 17 14:37:48 2016 -0800
Committer: Thomas Groh 
Committed: Thu Nov 17 14:37:48 2016 -0800

--
 .../direct/BoundedReadEvaluatorFactoryTest.java | 26 +++-
 1 file changed, 14 insertions(+), 12 deletions(-)
--

[26/50] incubator-beam git commit: Remove dependency on findbugs annotations artifact

[kenn]

Remove dependency on findbugs annotations artifact

The findbugs annotations artifact has an incompatible license and cannot
be included within Beam. We were previously referencing it for the
@SuppressFBWarning annotation for inline FindBugs suppression.

This change moves inline suppressions out to the existing
findbugs-filter.xml file. While not ideal as it removes the suppressions
from the offending context, it allows us to drop our dependency on the
incompatible artifact.

We are also referencing the @Nullable attribute from findbugs. This is
now source from findbugs jsr305, which does have a compatible license.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d39471b5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d39471b5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d39471b5

Branch: refs/heads/gearpump-runner
Commit: d39471b51e8d7a56aab2324adc4dbc2d70d9ab40
Parents: cb5d216
Author: Scott Wegner 
Authored: Mon Nov 21 13:07:46 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 14:29:35 2016 -0800

--
 pom.xml |  34 +---
 runners/apex/pom.xml|   2 +-
 runners/core-java/pom.xml   |   2 +-
 .../beam/runners/core/PaneInfoTracker.java  |   3 -
 .../beam/runners/core/SystemReduceFn.java   |   3 -
 .../apache/beam/runners/core/WatermarkHold.java |   3 -
 .../AfterDelayFromFirstElementStateMachine.java |   5 -
 .../core/triggers/AfterPaneStateMachine.java|   3 -
 .../core/triggers/TriggerStateMachines.java |  15 --
 runners/direct-java/pom.xml |   2 +-
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../src/main/resources/beam/findbugs-filter.xml | 172 ++-
 sdks/java/core/pom.xml  |   2 +-
 .../org/apache/beam/sdk/coders/AvroCoder.java   |   5 -
 .../apache/beam/sdk/coders/InstantCoder.java|   4 -
 .../org/apache/beam/sdk/testing/PAssert.java|   3 -
 .../beam/sdk/testing/SerializableMatchers.java  |   3 -
 ...AttemptAndTimeBoundedExponentialBackOff.java |   3 -
 .../sdk/util/ExposedByteArrayInputStream.java   |   2 -
 .../sdk/util/ExposedByteArrayOutputStream.java  |   3 -
 .../java/org/apache/beam/sdk/util/ZipFiles.java |   9 -
 .../beam/sdk/util/state/StateMerging.java   |   3 -
 sdks/java/io/google-cloud-platform/pom.xml  |  14 +-
 sdks/java/io/hdfs/pom.xml   |   2 +-
 sdks/java/io/jdbc/pom.xml   |   2 +-
 sdks/java/io/jms/pom.xml|   2 +-
 sdks/java/io/kafka/pom.xml  |   2 +-
 sdks/java/io/kinesis/pom.xml|   2 +-
 sdks/java/io/mongodb/pom.xml|   2 +-
 29 files changed, 178 insertions(+), 131 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d39471b5/pom.xml
--
diff --git a/pom.xml b/pom.xml
index bd9448a..e0ef47c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -390,12 +390,6 @@
 io.grpc
 grpc-core
 ${grpc.version}
-
-  
-com.google.code.findbugs
-jsr305
-  
-
   
 
   
@@ -473,10 +467,6 @@
 com.google.guava
 guava-jdk5
   
-  
-com.google.code.findbugs
-jsr305
-  
 
   
 
@@ -501,10 +491,6 @@
 com.google.guava
 guava-jdk5
   
-  
-com.google.code.findbugs
-jsr305
-  
 
   
 
@@ -664,7 +650,7 @@
 
   
 com.google.code.findbugs
-annotations
+jsr305
 ${findbugs.version}
   
 
@@ -672,24 +658,12 @@
 com.google.cloud.bigdataoss
 gcsio
 ${google-cloud-bigdataoss.version}
-
-  
-com.google.code.findbugs
-jsr305
-  
-
   
 
   
 com.google.cloud.bigdataoss
 util
 ${google-cloud-bigdataoss.version}
-
-  
-com.google.code.findbugs
-jsr305
-  
-
   
 
   
@@ -829,12 +803,6 @@
 guava-testlib
 ${guava.version}
 test
-
-  
-com.google.code.findbugs
-jsr305
-  
-
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d39471b5/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 1ca61b9..5478b24 100644
--- a/runners/apex/pom.xml
+++ b/runners/apex/pom.xml
@@ -101,7 +101,7 @@
 

  

[33/50] incubator-beam git commit: Reject stateful DoFn in SparkRunner

[kenn]

Reject stateful DoFn in SparkRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b0d07d74
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b0d07d74
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b0d07d74

Branch: refs/heads/gearpump-runner
Commit: b0d07d74f7805ee1d30fdedf54c089790d63d898
Parents: 8d71568
Author: Kenneth Knowles 
Authored: Tue Nov 15 21:33:13 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 21 21:34:20 2016 -0800

--
 runners/spark/pom.xml   |  1 +
 .../spark/translation/TransformTranslator.java  | 23 
 2 files changed, 24 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b0d07d74/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index 4c5b3f5..88223e2 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -72,6 +72,7 @@
 
 
   
org.apache.beam.sdk.testing.RunnableOnService
+  
org.apache.beam.sdk.testing.UsesStatefulParDo
   1
   false
   true

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b0d07d74/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
index c902ee3..60d668e 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
@@ -31,6 +31,7 @@ import org.apache.avro.mapred.AvroKey;
 import org.apache.avro.mapreduce.AvroJob;
 import org.apache.avro.mapreduce.AvroKeyInputFormat;
 import org.apache.beam.runners.core.AssignWindowsDoFn;
+import org.apache.beam.runners.spark.SparkRunner;
 import org.apache.beam.runners.spark.aggregators.AccumulatorSingleton;
 import org.apache.beam.runners.spark.aggregators.NamedAggregators;
 import org.apache.beam.runners.spark.io.SourceRDD;
@@ -47,12 +48,14 @@ import org.apache.beam.sdk.io.TextIO;
 import org.apache.beam.sdk.transforms.Combine;
 import org.apache.beam.sdk.transforms.CombineWithContext;
 import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
@@ -225,6 +228,16 @@ public final class TransformTranslator {
 return new TransformEvaluator>() {
   @Override
   public void evaluate(ParDo.Bound transform, 
EvaluationContext context) {
+DoFn doFn = transform.getNewFn();
+if 
(DoFnSignatures.getSignature(doFn.getClass()).stateDeclarations().size() > 0) {
+  throw new UnsupportedOperationException(
+  String.format(
+  "Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
+  DoFn.StateId.class.getSimpleName(),
+  doFn.getClass().getName(),
+  DoFn.class.getSimpleName(),
+  SparkRunner.class.getSimpleName()));
+}
 @SuppressWarnings("unchecked")
 JavaRDD> inRDD =
 ((BoundedDataset) 
context.borrowDataset(transform)).getRDD();
@@ -247,6 +260,16 @@ public final class TransformTranslator {
 return new TransformEvaluator>() {
   @Override
   public void evaluate(ParDo.BoundMulti transform, 
EvaluationContext context) {
+DoFn doFn = transform.getNewFn();
+if 
(DoFnSignatures.getSignature(doFn.getClass()).stateDeclarations().size() > 0) {
+  throw new UnsupportedOperationException(
+  String.format(
+  "Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
+  DoFn.StateId.class.getSimpleName(),
+  doFn.getClass().getName(),
+  DoFn.class.getSimpleName(),
+  SparkRunner.class.getSimpleName()));
+}
 @SuppressWarnings

[41/50] incubator-beam git commit: This closes #1406

[kenn]

This closes #1406


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c2dc3863
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c2dc3863
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c2dc3863

Branch: refs/heads/gearpump-runner
Commit: c2dc38639ddd37c393d3bd9f341c2ec032a1096c
Parents: e53d6d4 c80554b
Author: Thomas Groh 
Authored: Tue Nov 22 10:10:02 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 22 10:10:02 2016 -0800

--
 .../src/main/java/StarterPipeline.java| 18 ++
 .../src/main/java/it/pkg/StarterPipeline.java | 18 ++
 2 files changed, 20 insertions(+), 16 deletions(-)
--

[34/50] incubator-beam git commit: Beam archetypes: enable snapshot repositories.

[kenn]

Beam archetypes: enable snapshot repositories.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f3c4709f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f3c4709f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f3c4709f

Branch: refs/heads/gearpump-runner
Commit: f3c4709f6a8d6260f98ccdb1492d8094d87dd696
Parents: 70efa47
Author: Pei He 
Authored: Mon Nov 21 17:19:27 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 23:38:11 2016 -0800

--
 .../src/main/resources/archetype-resources/pom.xml| 14 ++
 .../src/main/resources/archetype-resources/pom.xml| 14 ++
 .../test/resources/projects/basic/reference/pom.xml   | 14 ++
 3 files changed, 42 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3c4709f/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 45bcf8a..b18c57c 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -26,6 +26,20 @@
 
   jar
 
+  
+
+  apache.snapshots
+  Apache Development Snapshot Repository
+  https://repository.apache.org/content/repositories/snapshots/
+  
+false
+  
+  
+true
+  
+
+  
+
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3c4709f/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
index 5287d1f..738acde 100644
--- 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
@@ -24,6 +24,20 @@
   ${artifactId}
   ${version}
 
+  
+
+  apache.snapshots
+  Apache Development Snapshot Repository
+  https://repository.apache.org/content/repositories/snapshots/
+  
+false
+  
+  
+true
+  
+
+  
+
   

   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3c4709f/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
index cce2f99..6950ed5 100644
--- 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
@@ -24,6 +24,20 @@
   basic
   0.1
 
+  
+
+  apache.snapshots
+  Apache Development Snapshot Repository
+  https://repository.apache.org/content/repositories/snapshots/
+  
+false
+  
+  
+true
+  
+
+  
+
   

   

[01/50] incubator-beam git commit: Restores StateContexts.stateContextFromComponents

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/gearpump-runner a14927f40 -> 28124050d


Restores StateContexts.stateContextFromComponents

It must be temporarily restored for compatibility with
current Dataflow worker in order for integration tests
to pass.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a23ae207
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a23ae207
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a23ae207

Branch: refs/heads/gearpump-runner
Commit: a23ae20760011634338e1c1eca9ec1891335fd6b
Parents: 90a0d0e
Author: Eugene Kirpichov 
Authored: Wed Nov 16 13:52:52 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Nov 17 13:18:36 2016 -0800

--
 .../beam/sdk/util/state/StateContexts.java  | 39 
 1 file changed, 39 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a23ae207/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
index 81121e1..858d6fe 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
@@ -17,8 +17,10 @@
  */
 package org.apache.beam.sdk.util.state;
 
+import javax.annotation.Nullable;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.values.PCollectionView;
 
 /**
@@ -49,4 +51,41 @@ public class StateContexts {
   public static  StateContext nullContext() {
 return (StateContext) NULL_CONTEXT;
   }
+
+  /**
+   * Deprecated, do not use.
+   *
+   * This exists only for temporary compatibility with Dataflow worker and 
should be deleted
+   * once a worker image is released that uses runners-core build after
+   * https://github.com/apache/incubator-beam/pull/1353.
+   */
+  @Deprecated
+  public static  StateContext createFromComponents(
+  @Nullable final PipelineOptions options,
+  final WindowingInternals windowingInternals,
+  final W window) {
+@SuppressWarnings("unchecked")
+StateContext typedNullContext = (StateContext) NULL_CONTEXT;
+if (options == null) {
+  return typedNullContext;
+} else {
+  return new StateContext() {
+
+@Override
+public PipelineOptions getPipelineOptions() {
+  return options;
+}
+
+@Override
+public  T sideInput(PCollectionView view) {
+  return windowingInternals.sideInput(view, window);
+}
+
+@Override
+public W window() {
+  return window;
+}
+  };
+}
+  }
 }

[08/50] incubator-beam git commit: Closes #1381

[kenn]

Closes #1381


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bb9c3866
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bb9c3866
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bb9c3866

Branch: refs/heads/gearpump-runner
Commit: bb9c3866421863b257470ec9fb6bcb8328158045
Parents: 4852d2e 5fc68a5
Author: bchambers 
Authored: Fri Nov 18 09:55:29 2016 -0800
Committer: bchambers 
Committed: Fri Nov 18 09:55:29 2016 -0800

--
 .../org/apache/beam/sdk/metrics/MetricsContainer.java  | 13 -
 1 file changed, 13 deletions(-)
--

[50/50] incubator-beam git commit: Merge branch 'master' of https://github.com/apache/incubator-beam into gearpump-runner

[kenn]

Merge branch 'master' of https://github.com/apache/incubator-beam into 
gearpump-runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/28124050
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/28124050
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/28124050

Branch: refs/heads/gearpump-runner
Commit: 28124050d5598acf8c9df621551ed2d84dc643fd
Parents: 68363d0 dd85cad
Author: manuzhang 
Authored: Wed Nov 23 12:32:37 2016 +0800
Committer: manuzhang 
Committed: Wed Nov 23 12:32:37 2016 +0800

--
 README.md   |   4 +-
 pom.xml |  34 +-
 runners/apex/pom.xml|   3 +-
 .../translation/ParDoBoundMultiTranslator.java  |  67 ++-
 .../apex/translation/ParDoBoundTranslator.java  |  46 +-
 .../operators/ApexGroupByKeyOperator.java   |  26 +-
 runners/core-java/pom.xml   |   2 +-
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   3 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   5 +-
 .../beam/runners/core/OutputWindowedValue.java  |  45 ++
 .../beam/runners/core/PaneInfoTracker.java  |   3 -
 .../runners/core/ReduceFnContextFactory.java|  83 +++-
 .../beam/runners/core/ReduceFnRunner.java   |  40 +-
 .../beam/runners/core/SimpleDoFnRunner.java | 142 --
 .../beam/runners/core/SimpleOldDoFnRunner.java  |  21 +-
 .../beam/runners/core/SplittableParDo.java  |  35 +-
 .../beam/runners/core/SystemReduceFn.java   |   3 -
 .../apache/beam/runners/core/WatermarkHold.java |   3 -
 .../core/WindowingInternalsAdapters.java|  76 
 .../AfterDelayFromFirstElementStateMachine.java |   5 -
 .../core/triggers/AfterPaneStateMachine.java|   3 -
 .../core/triggers/TriggerStateMachines.java |  15 -
 .../beam/runners/core/ReduceFnTester.java   |  81 +---
 runners/direct-java/pom.xml |   2 +-
 .../direct/BoundedReadEvaluatorFactory.java | 110 -
 .../beam/runners/direct/DirectRunner.java   |  18 +-
 .../beam/runners/direct/EmptyInputProvider.java |  22 +-
 .../direct/ExecutorServiceParallelExecutor.java |  12 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |  92 ++--
 .../runners/direct/ParDoEvaluatorFactory.java   |  56 +--
 .../direct/ParDoMultiEvaluatorHooks.java|  55 ---
 .../direct/ParDoMultiOverrideFactory.java   |  51 +++
 .../runners/direct/ParDoOverrideFactory.java|  53 ---
 .../direct/ParDoSingleEvaluatorHooks.java   |  58 ---
 .../ParDoSingleViaMultiOverrideFactory.java |  66 +++
 .../beam/runners/direct/RootInputProvider.java  |  16 +-
 .../runners/direct/RootProviderRegistry.java|  19 +-
 .../runners/direct/StepTransformResult.java |   6 +
 .../direct/TestStreamEvaluatorFactory.java  |  23 +-
 .../direct/TransformEvaluatorRegistry.java  |  12 +-
 .../beam/runners/direct/TransformExecutor.java  |  28 +-
 .../direct/UnboundedReadEvaluatorFactory.java   |  32 +-
 .../direct/BoundedReadEvaluatorFactoryTest.java | 186 +++-
 .../direct/ParDoMultiEvaluatorHooksTest.java| 439 ---
 .../direct/ParDoSingleEvaluatorHooksTest.java   | 335 --
 .../runners/direct/TransformExecutorTest.java   |  43 --
 runners/flink/runner/pom.xml|   1 +
 .../FlinkBatchTransformTranslators.java |  34 +-
 .../FlinkStreamingTransformTranslators.java |  38 +-
 .../functions/FlinkDoFnFunction.java|  21 +-
 .../FlinkMergingNonShuffleReduceFunction.java   |  14 +-
 .../FlinkMergingPartialReduceFunction.java  |  14 +-
 .../functions/FlinkMergingReduceFunction.java   |  12 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |  14 +-
 .../FlinkMultiOutputProcessContext.java |  94 +---
 .../functions/FlinkPartialReduceFunction.java   |  14 +-
 .../functions/FlinkProcessContext.java  | 330 --
 .../functions/FlinkProcessContextBase.java  | 278 
 .../functions/FlinkReduceFunction.java  |  14 +-
 .../FlinkSingleOutputProcessContext.java|  69 +++
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../beam/runners/dataflow/DataflowRunner.java   |   2 +-
 .../dataflow/util/DataflowTransport.java|  10 +-
 .../options/DataflowPipelineOptionsTest.java|   6 +-
 .../runners/dataflow/util/PackageUtilTest.java  |   2 +-
 runners/spark/pom.xml   |   4 +-
 .../runners/spark/SparkPipelineOptions.java |  10 +-
 .../apache/beam/runners/spark/SparkRunner.java  |   9 +-
 .../spark/translation/BoundedDataset.java   | 114 +
 .../beam/runners/spark/translation/Dataset.java |  34 ++
 .../runners/spark/translation/DoFnFunction.java |  10 +-
 .../spark/translation/EvaluationContext.java| 257 +--
 .../spark/translation/MultiDoFnFunction.java  

[13/50] incubator-beam git commit: This closes #1389

[kenn]

This closes #1389


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3548ffb0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3548ffb0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3548ffb0

Branch: refs/heads/gearpump-runner
Commit: 3548ffb0656c4729fc1475490cc7f0e21669c197
Parents: 2a7169b 73692b8
Author: Kenneth Knowles 
Authored: Fri Nov 18 15:00:13 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:00:13 2016 -0800

--
 .../beam/sdk/util/state/StateContexts.java  | 41 +++-
 1 file changed, 32 insertions(+), 9 deletions(-)
--

[14/50] incubator-beam git commit: This closes #1378

[kenn]

This closes #1378


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d93e9a88
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d93e9a88
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d93e9a88

Branch: refs/heads/gearpump-runner
Commit: d93e9a88b0f0b3c7dc2461cbd11147b9ad8327cf
Parents: 3548ffb f86e98c
Author: Kenneth Knowles 
Authored: Fri Nov 18 15:09:43 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../beam/runners/direct/DirectRunner.java   |   3 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |  56 +--
 .../direct/ParDoMultiEvaluatorHooks.java|  55 ---
 .../direct/ParDoMultiOverrideFactory.java   |  51 +++
 .../runners/direct/ParDoOverrideFactory.java|  53 ---
 .../direct/ParDoSingleEvaluatorHooks.java   |  58 ---
 .../ParDoSingleViaMultiOverrideFactory.java |  66 +++
 .../direct/TransformEvaluatorRegistry.java  |   7 +-
 .../direct/ParDoMultiEvaluatorHooksTest.java| 439 ---
 .../direct/ParDoSingleEvaluatorHooksTest.java   | 335 --
 .../org/apache/beam/sdk/transforms/DoFn.java|   7 +-
 .../beam/sdk/transforms/DoFnAdapters.java   |   6 +
 .../org/apache/beam/sdk/transforms/OldDoFn.java |   7 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |   2 +-
 .../beam/sdk/transforms/WithTimestamps.java |   5 +-
 .../apache/beam/sdk/transforms/OldDoFnTest.java |  11 +-
 .../apache/beam/sdk/transforms/ParDoTest.java   |  55 ++-
 17 files changed, 194 insertions(+), 1022 deletions(-)
--

[35/50] incubator-beam git commit: This closes #1407

[kenn]

This closes #1407


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/249dbc04
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/249dbc04
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/249dbc04

Branch: refs/heads/gearpump-runner
Commit: 249dbc045ccfbfb19280114e973c52d48218839d
Parents: 70efa47 f3c4709
Author: Davor Bonaci 
Authored: Mon Nov 21 23:38:19 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 23:38:19 2016 -0800

--
 .../src/main/resources/archetype-resources/pom.xml| 14 ++
 .../src/main/resources/archetype-resources/pom.xml| 14 ++
 .../test/resources/projects/basic/reference/pom.xml   | 14 ++
 3 files changed, 42 insertions(+)
--

[36/50] incubator-beam git commit: [BEAM-59] Drops public constructors and uses Factory methods in Gcs/File/IOChannelFactory.

[kenn]

[BEAM-59] Drops public constructors and uses Factory methods in 
Gcs/File/IOChannelFactory.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e6fa2ff2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e6fa2ff2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e6fa2ff2

Branch: refs/heads/gearpump-runner
Commit: e6fa2ff26e836848fa549c290ed098dd019cc4e1
Parents: 249dbc0
Author: Pei He 
Authored: Mon Oct 31 17:58:31 2016 -0700
Committer: Luke Cwik 
Committed: Tue Nov 22 06:18:54 2016 -0800

--
 .../apache/beam/sdk/util/FileIOChannelFactory.java | 17 +
 .../apache/beam/sdk/util/GcsIOChannelFactory.java  | 10 +-
 .../org/apache/beam/sdk/util/IOChannelUtils.java   |  7 +++
 .../beam/sdk/util/FileIOChannelFactoryTest.java|  2 +-
 .../beam/sdk/util/GcsIOChannelFactoryTest.java |  2 +-
 5 files changed, 31 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e6fa2ff2/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java
index 0eefb77..13591a3 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactory.java
@@ -44,6 +44,7 @@ import java.util.Collections;
 import java.util.LinkedList;
 import java.util.List;
 import java.util.regex.Matcher;
+import org.apache.beam.sdk.options.PipelineOptions;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -53,6 +54,22 @@ import org.slf4j.LoggerFactory;
 public class FileIOChannelFactory implements IOChannelFactory {
   private static final Logger LOG = 
LoggerFactory.getLogger(FileIOChannelFactory.class);
 
+   /**
+   * Create a {@link FileIOChannelFactory} with the given {@link 
PipelineOptions}.
+   */
+  public static FileIOChannelFactory fromOptions(PipelineOptions options) {
+return create();
+  }
+
+  /**
+   * Create a {@link FileIOChannelFactory}.
+   */
+  public static FileIOChannelFactory create() {
+return new FileIOChannelFactory();
+  }
+
+  private FileIOChannelFactory() {}
+
   /**
*  Converts the given file spec to a java {@link File}. If {@code spec} is 
actually a URI with
*  the {@code file} scheme, then this function will ensure that the 
returned {@link File}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e6fa2ff2/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactory.java
index bd2ec4e..9f99cd6 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactory.java
@@ -25,6 +25,7 @@ import java.util.Collection;
 import java.util.LinkedList;
 import java.util.List;
 import org.apache.beam.sdk.options.GcsOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.util.gcsfs.GcsPath;
 
 /**
@@ -32,9 +33,16 @@ import org.apache.beam.sdk.util.gcsfs.GcsPath;
  */
 public class GcsIOChannelFactory implements IOChannelFactory {
 
+  /**
+   * Create a {@link GcsIOChannelFactory} with the given {@link 
PipelineOptions}.
+   */
+  public static GcsIOChannelFactory fromOptions(PipelineOptions options) {
+return new GcsIOChannelFactory(options.as(GcsOptions.class));
+  }
+
   private final GcsOptions options;
 
-  public GcsIOChannelFactory(GcsOptions options) {
+  private GcsIOChannelFactory(GcsOptions options) {
 this.options = options;
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e6fa2ff2/sdks/java/core/src/main/java/org/apache/beam/sdk/util/IOChannelUtils.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/IOChannelUtils.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/IOChannelUtils.java
index 16a6e95..d221fa9 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/IOChannelUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/IOChannelUtils.java
@@ -29,7 +29,6 @@ import java.util.Map;
 import java.util.Set;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
-import org.apache.beam.sdk.options.GcsOptions;
 import org.apache.beam.sdk.options.Pi

[38/50] incubator-beam git commit: [BEAM-59] Use ServiceLoader to register IOChannelFactories in IOChannelUtils.

[kenn]

[BEAM-59] Use ServiceLoader to register IOChannelFactories in IOChannelUtils.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fa417f9c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fa417f9c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fa417f9c

Branch: refs/heads/gearpump-runner
Commit: fa417f9c2c671626eba3326e82d47741000ec64d
Parents: cd1a5e7
Author: Pei He 
Authored: Mon Oct 31 18:02:49 2016 -0700
Committer: Luke Cwik 
Committed: Tue Nov 22 06:18:55 2016 -0800

--
 .../beam/runners/dataflow/DataflowRunner.java   |   2 +-
 .../options/DataflowPipelineOptionsTest.java|   6 +-
 .../runners/dataflow/util/PackageUtilTest.java  |   2 +-
 .../sdk/options/PipelineOptionsFactory.java |  32 +
 .../apache/beam/sdk/runners/PipelineRunner.java |   2 +-
 .../apache/beam/sdk/testing/TestPipeline.java   |   2 +-
 .../beam/sdk/util/FileIOChannelFactory.java |  10 +-
 .../sdk/util/IOChannelFactoryRegistrar.java |  11 +-
 .../apache/beam/sdk/util/IOChannelUtils.java| 133 ++-
 .../beam/sdk/util/common/ReflectHelpers.java|  29 
 .../java/org/apache/beam/sdk/io/AvroIOTest.java |   2 +-
 .../apache/beam/sdk/io/FileBasedSourceTest.java |   2 +-
 .../java/org/apache/beam/sdk/io/TextIOTest.java |   2 +-
 .../sdk/options/PipelineOptionsFactoryTest.java |  34 -
 .../util/FileIOChannelFactoryRegistrarTest.java |   4 +-
 .../beam/sdk/util/FileIOChannelFactoryTest.java |   2 +-
 .../util/GcsIOChannelFactoryRegistrarTest.java  |   4 +-
 .../beam/sdk/util/IOChannelUtilsTest.java   |  39 ++
 .../sdk/util/common/ReflectHelpersTest.java |  33 +
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java |   6 +-
 20 files changed, 259 insertions(+), 98 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fa417f9c/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 841b13f..36328e9 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -240,7 +240,7 @@ public class DataflowRunner extends 
PipelineRunner {
*/
   public static DataflowRunner fromOptions(PipelineOptions options) {
 // (Re-)register standard IO factories. Clobbers any prior credentials.
-IOChannelUtils.registerStandardIOFactories(options);
+IOChannelUtils.registerIOFactoriesAllowOverride(options);
 
 DataflowPipelineOptions dataflowOptions =
 PipelineOptionsValidator.validate(DataflowPipelineOptions.class, 
options);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fa417f9c/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
index 202d04b..52082e0 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptionsTest.java
@@ -126,7 +126,7 @@ public class DataflowPipelineOptionsTest {
   @Test
   public void testStagingLocation() {
 DataflowPipelineOptions options = 
PipelineOptionsFactory.as(DataflowPipelineOptions.class);
-IOChannelUtils.registerStandardIOFactories(options);
+IOChannelUtils.registerIOFactoriesAllowOverride(options);
 options.setTempLocation("file://temp_location");
 options.setStagingLocation("gs://staging_location");
 assertTrue(isNullOrEmpty(options.getGcpTempLocation()));
@@ -136,7 +136,7 @@ public class DataflowPipelineOptionsTest {
   @Test
   public void testDefaultToTempLocation() {
 DataflowPipelineOptions options = 
PipelineOptionsFactory.as(DataflowPipelineOptions.class);
-IOChannelUtils.registerStandardIOFactories(options);
+IOChannelUtils.registerIOFactoriesAllowOverride(options);
 options.setPathValidatorClass(NoopPathValidator.class);
 options.setTempLocation("gs://temp_location");
 assertEquals("gs://temp_location", option

[06/50] incubator-beam git commit: Added messages to exceptions

[kenn]

Added messages to exceptions


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c0623c10
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c0623c10
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c0623c10

Branch: refs/heads/gearpump-runner
Commit: c0623c10318f95984747bd94fd5d733be5e5de11
Parents: a23ae20
Author: Eugene Kirpichov 
Authored: Thu Nov 17 15:36:12 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Nov 17 15:36:12 2016 -0800

--
 .../beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c0623c10/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
index a5bb214..b946e4d 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
@@ -182,12 +182,14 @@ class GroupAlsoByWindowEvaluatorFactory implements 
TransformEvaluatorFactory {
 
 @Override
 public  boolean contains(PCollectionView view) {
-  throw new UnsupportedOperationException();
+  throw new UnsupportedOperationException(
+  "GroupAlsoByWindow must not have side inputs");
 }
 
 @Override
 public boolean isEmpty() {
-  throw new UnsupportedOperationException();
+  throw new UnsupportedOperationException(
+  "GroupAlsoByWindow must not have side inputs");
 }
   },
   droppedDueToClosedWindow,

[07/50] incubator-beam git commit: This closes #1353

[kenn]

This closes #1353


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4852d2eb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4852d2eb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4852d2eb

Branch: refs/heads/gearpump-runner
Commit: 4852d2ebf503a5207db612cb6429728d0f3cb7e1
Parents: 212fec4 c0623c1
Author: Kenneth Knowles 
Authored: Thu Nov 17 22:31:07 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Nov 17 22:31:07 2016 -0800

--
 .../operators/ApexGroupByKeyOperator.java   |  26 +-
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   3 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   5 +-
 .../beam/runners/core/OutputWindowedValue.java  |  45 +++
 .../runners/core/ReduceFnContextFactory.java|  83 -
 .../beam/runners/core/ReduceFnRunner.java   |  40 +--
 .../beam/runners/core/SimpleDoFnRunner.java |  42 ++-
 .../beam/runners/core/SimpleOldDoFnRunner.java  |  21 +-
 .../core/WindowingInternalsAdapters.java|  76 +
 .../beam/runners/core/ReduceFnTester.java   |  81 ++---
 .../GroupAlsoByWindowEvaluatorFactory.java  |  88 ++---
 .../functions/FlinkDoFnFunction.java|  21 +-
 .../FlinkMergingNonShuffleReduceFunction.java   |  14 +-
 .../FlinkMergingPartialReduceFunction.java  |  14 +-
 .../functions/FlinkMergingReduceFunction.java   |  12 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |  14 +-
 .../FlinkMultiOutputProcessContext.java |  94 ++
 .../functions/FlinkPartialReduceFunction.java   |  14 +-
 .../functions/FlinkProcessContext.java  | 330 ---
 .../functions/FlinkProcessContextBase.java  | 278 
 .../functions/FlinkReduceFunction.java  |  14 +-
 .../FlinkSingleOutputProcessContext.java|  69 
 .../runners/spark/translation/DoFnFunction.java |  10 +-
 .../spark/translation/MultiDoFnFunction.java|  21 +-
 .../spark/translation/SparkProcessContext.java  |  81 +++--
 .../apache/beam/sdk/transforms/DoFnTester.java  |  18 +-
 .../beam/sdk/util/WindowingInternals.java   |  14 +-
 .../beam/sdk/util/state/StateContexts.java  |  29 +-
 28 files changed, 827 insertions(+), 730 deletions(-)
--

[11/50] incubator-beam git commit: This closes #1386

[kenn]

This closes #1386


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2a7169b6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2a7169b6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2a7169b6

Branch: refs/heads/gearpump-runner
Commit: 2a7169b6f62de765fe1e4b28bad195f5e102aec7
Parents: bb9c386 33fb8c2
Author: Kenneth Knowles 
Authored: Fri Nov 18 14:20:21 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 14:20:21 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java |   4 +-
 .../beam/runners/core/SplittableParDo.java  |   7 +-
 .../org/apache/beam/sdk/transforms/DoFn.java| 122 ---
 .../beam/sdk/transforms/DoFnAdapters.java   |  10 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java|  41 +++
 .../sdk/transforms/reflect/DoFnInvoker.java | 121 +-
 .../sdk/transforms/reflect/DoFnInvokers.java|   4 +-
 .../sdk/transforms/reflect/OnTimerInvoker.java  |   8 +-
 .../transforms/reflect/DoFnInvokersTest.java|   5 +-
 .../transforms/reflect/OnTimerInvokersTest.java |   2 +-
 .../transforms/DoFnInvokersBenchmark.java   |   5 +-
 11 files changed, 161 insertions(+), 168 deletions(-)
--

[30/50] incubator-beam git commit: Add JUnit category for stateful ParDo tests

[kenn]

Add JUnit category for stateful ParDo tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f8b6bb7f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f8b6bb7f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f8b6bb7f

Branch: refs/heads/gearpump-runner
Commit: f8b6bb7f1ab8720ca4f2d766831d8f243dd27085
Parents: 70efa47
Author: Kenneth Knowles 
Authored: Mon Nov 21 15:41:13 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 21 21:32:47 2016 -0800

--
 .../beam/sdk/testing/UsesStatefulParDo.java | 25 
 1 file changed, 25 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f8b6bb7f/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStatefulParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStatefulParDo.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStatefulParDo.java
new file mode 100644
index 000..8bd6330
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStatefulParDo.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+import org.apache.beam.sdk.transforms.ParDo;
+
+/**
+ * Category tag for validation tests which utilize stateful {@link ParDo}.
+ */
+public interface UsesStatefulParDo {}

[42/50] incubator-beam git commit: This closes #1409

[kenn]

This closes #1409


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7949b708
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7949b708
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7949b708

Branch: refs/heads/gearpump-runner
Commit: 7949b708257c6e1dbd86db99bb5e10e2e47d33aa
Parents: c2dc386 b0d07d7
Author: Kenneth Knowles 
Authored: Tue Nov 22 10:51:14 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 10:51:14 2016 -0800

--
 runners/spark/pom.xml   |  1 +
 .../spark/translation/TransformTranslator.java  | 23 ++
 .../beam/sdk/testing/UsesStatefulParDo.java | 25 
 3 files changed, 49 insertions(+)
--

[16/50] incubator-beam git commit: Use getNewFn for coder inferences in ParDo

[kenn]

Use getNewFn for coder inferences in ParDo


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/34e2a352
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/34e2a352
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/34e2a352

Branch: refs/heads/gearpump-runner
Commit: 34e2a3525643f98ddcee5889b2d0d232e2e05509
Parents: b0d46c2
Author: Kenneth Knowles 
Authored: Thu Nov 17 11:08:48 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/34e2a352/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index ac1bccb..215ae6a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -1094,7 +1094,7 @@ public class ParDo {
   Coder inputCoder = ((PCollection) input).getCoder();
   return input.getPipeline().getCoderRegistry().getDefaultCoder(
   output.getTypeDescriptor(),
-  getOldFn().getInputTypeDescriptor(),
+  getNewFn().getInputTypeDescriptor(),
   inputCoder);
   }
 

[03/50] incubator-beam git commit: Refactor SparkProcessContext more cleanly into single- and multi-output versions

[kenn]

Refactor SparkProcessContext more cleanly into single- and multi-output versions


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8243fcdc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8243fcdc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8243fcdc

Branch: refs/heads/gearpump-runner
Commit: 8243fcdc4e80838589622bd2f0bbe51350da8c8a
Parents: 1fb1f7b
Author: Eugene Kirpichov 
Authored: Fri Nov 11 16:57:27 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Nov 17 13:18:36 2016 -0800

--
 .../beam/runners/core/OutputWindowedValue.java  |  1 -
 .../runners/core/ReduceFnContextFactory.java|  1 -
 .../runners/spark/translation/DoFnFunction.java |  9 +++-
 .../spark/translation/MultiDoFnFunction.java| 17 +-
 .../spark/translation/SparkProcessContext.java  | 57 ++--
 5 files changed, 39 insertions(+), 46 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8243fcdc/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputWindowedValue.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputWindowedValue.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputWindowedValue.java
index 08a0e81..86eeb33 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputWindowedValue.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/OutputWindowedValue.java
@@ -20,7 +20,6 @@ package org.apache.beam.runners.core;
 import java.util.Collection;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.values.TupleTag;
 import org.joda.time.Instant;
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8243fcdc/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
index 668ef47..d43fb8e 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
@@ -33,7 +33,6 @@ import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.TimerInternals.TimerData;
 import org.apache.beam.sdk.util.Timers;
-import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.MergingStateAccessor;
 import org.apache.beam.sdk.util.state.ReadableState;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8243fcdc/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
index 4dfbee6..fa08c5b 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
@@ -32,7 +32,7 @@ import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.TupleTag;
 import org.apache.spark.Accumulator;
 import org.apache.spark.api.java.function.FlatMapFunction;
-
+import org.joda.time.Instant;
 
 
 /**
@@ -93,6 +93,13 @@ public class DoFnFunction
 }
 
 @Override
+public  void sideOutput(TupleTag tag, WindowedValue output) {
+  throw new UnsupportedOperationException(
+  "sideOutput is an unsupported operation for doFunctions, use a "
+  + "MultiDoFunction instead.");
+}
+
+@Override
 public Accumulator getAccumulator() {
   return accum;
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8243fcdc/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/MultiDoFnFunction.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/MultiDoFnFunction.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/MultiDoFnFunction.java
index 1168381..d015b08 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/M

[37/50] incubator-beam git commit: [BEAM-59] Create IOChannelFactoryRegistrar interface and its gcs/file implementations.

[kenn]

[BEAM-59] Create IOChannelFactoryRegistrar interface and its gcs/file 
implementations.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cd1a5e7e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cd1a5e7e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cd1a5e7e

Branch: refs/heads/gearpump-runner
Commit: cd1a5e7e30a3bd46f822d371333afa975fc7e4af
Parents: e6fa2ff
Author: Pei He 
Authored: Mon Oct 31 18:01:41 2016 -0700
Committer: Luke Cwik 
Committed: Tue Nov 22 06:18:54 2016 -0800

--
 .../sdk/util/FileIOChannelFactoryRegistrar.java | 38 +
 .../sdk/util/GcsIOChannelFactoryRegistrar.java  | 38 +
 .../sdk/util/IOChannelFactoryRegistrar.java | 43 +++
 .../util/FileIOChannelFactoryRegistrarTest.java | 44 
 .../util/GcsIOChannelFactoryRegistrarTest.java  | 44 
 5 files changed, 207 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cd1a5e7e/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactoryRegistrar.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactoryRegistrar.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactoryRegistrar.java
new file mode 100644
index 000..acc0222
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/FileIOChannelFactoryRegistrar.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.options.PipelineOptions;
+
+/**
+ * {@link AutoService} registrar for the {@link FileIOChannelFactory}.
+ */
+@AutoService(IOChannelFactoryRegistrar.class)
+public class FileIOChannelFactoryRegistrar implements 
IOChannelFactoryRegistrar {
+
+  @Override
+  public IOChannelFactory fromOptions(PipelineOptions options) {
+return FileIOChannelFactory.fromOptions(options);
+  }
+
+  @Override
+  public String getScheme() {
+return "file";
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cd1a5e7e/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactoryRegistrar.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactoryRegistrar.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactoryRegistrar.java
new file mode 100644
index 000..b4c457f
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/GcsIOChannelFactoryRegistrar.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import com.google.auto.service.AutoService;
+import org.apache.beam.sdk.options.PipelineOptions;
+
+/**
+ * {@link AutoService} registrar for the {@link GcsIOChannelFactory}.
+ */
+@AutoService(IOChannelFactoryRegistrar.class)
+public class GcsIOChannelFactoryRegistrar implements IOChannelFactoryRegistrar 
{
+
+  @Override
+  public GcsIOChannelFactory fromOptions(PipelineOptions options) {
+return G

[21/50] incubator-beam git commit: This closes #1393

[kenn]

This closes #1393


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/875631f0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/875631f0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/875631f0

Branch: refs/heads/gearpump-runner
Commit: 875631f07b1e4556afec28dc850bd7fe2d07444b
Parents: d93e9a8 dafd5be
Author: Sela 
Authored: Sun Nov 20 19:26:52 2016 +0200
Committer: Sela 
Committed: Sun Nov 20 19:26:52 2016 +0200

--
 .../runners/spark/SparkPipelineOptions.java |  6 --
 .../apache/beam/runners/spark/SparkRunner.java  |  5 +-
 .../spark/translation/EvaluationContext.java| 59 +---
 .../SparkRunnerStreamingContextFactory.java |  3 +-
 .../streaming/EmptyStreamAssertionTest.java |  3 +-
 .../streaming/FlattenStreamingTest.java |  4 +-
 .../streaming/KafkaStreamingTest.java   | 11 ++--
 .../ResumeFromCheckpointStreamingTest.java  |  3 +-
 .../streaming/SimpleStreamingWordCountTest.java |  4 +-
 .../streaming/utils/PAssertStreaming.java   |  8 ++-
 .../SparkTestPipelineOptionsForStreaming.java   |  1 -
 11 files changed, 60 insertions(+), 47 deletions(-)
--

[44/50] incubator-beam git commit: This closes #1408

[kenn]

This closes #1408


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e7d7aa93
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e7d7aa93
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e7d7aa93

Branch: refs/heads/gearpump-runner
Commit: e7d7aa938b815ffac04f25942d27db4986a0368d
Parents: 7949b70 9c2d5da
Author: Kenneth Knowles 
Authored: Tue Nov 22 10:53:09 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 10:53:09 2016 -0800

--
 runners/flink/runner/pom.xml|  1 +
 .../FlinkBatchTransformTranslators.java | 34 +---
 .../FlinkStreamingTransformTranslators.java | 25 +-
 3 files changed, 55 insertions(+), 5 deletions(-)
--

[22/50] incubator-beam git commit: [BEAM-1018] Updated getEstimatedSizeBytes() to use Number.longValue() in MongoDbIO

[kenn]

[BEAM-1018] Updated getEstimatedSizeBytes() to use Number.longValue() in 
MongoDbIO


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a761b0c1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a761b0c1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a761b0c1

Branch: refs/heads/gearpump-runner
Commit: a761b0c154b67daada2968c24c7de9c4afe13884
Parents: 875631f
Author: Sandeep Parikh 
Authored: Mon Nov 21 10:05:36 2016 -0600
Committer: Jean-Baptiste Onofré 
Committed: Mon Nov 21 20:37:11 2016 +0100

--
 .../src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java| 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a761b0c1/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
--
diff --git 
a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
 
b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
index 71c017d..cd5cdaf 100644
--- 
a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
+++ 
b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
@@ -224,7 +224,7 @@ public class MongoDbIO {
   BasicDBObject stat = new BasicDBObject();
   stat.append("collStats", spec.collection());
   Document stats = mongoDatabase.runCommand(stat);
-  return Long.parseLong(stats.get("size").toString());
+  return stats.get("size", Number.class).longValue();
 }
 
 @Override

[31/50] incubator-beam git commit: Reject stateful DoFn in ApexRunner

[kenn]

Reject stateful DoFn in ApexRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e85cea78
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e85cea78
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e85cea78

Branch: refs/heads/gearpump-runner
Commit: e85cea78253d2f316a18d95d65aabc1176448841
Parents: f8b6bb7
Author: Kenneth Knowles 
Authored: Tue Nov 15 21:33:01 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 21 21:32:47 2016 -0800

--
 runners/apex/pom.xml|  1 +
 .../translation/ParDoBoundMultiTranslator.java  | 67 +---
 .../apex/translation/ParDoBoundTranslator.java  | 46 +-
 3 files changed, 74 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e85cea78/runners/apex/pom.xml
--
diff --git a/runners/apex/pom.xml b/runners/apex/pom.xml
index 5478b24..d0b0fdf 100644
--- a/runners/apex/pom.xml
+++ b/runners/apex/pom.xml
@@ -185,6 +185,7 @@
 
 
   org.apache.beam.sdk.testing.RunnableOnService
+  
org.apache.beam.sdk.testing.UsesStatefulParDo
   none
   true
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e85cea78/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
index 7c91b91..fed5f4b 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
@@ -23,17 +23,17 @@ import static 
com.google.common.base.Preconditions.checkArgument;
 import com.datatorrent.api.Operator;
 import com.datatorrent.api.Operator.OutputPort;
 import com.google.common.collect.Maps;
-
 import java.util.ArrayList;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
-
+import org.apache.beam.runners.apex.ApexRunner;
 import org.apache.beam.runners.apex.translation.operators.ApexParDoOperator;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
 import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
 import org.apache.beam.sdk.values.PCollection;
@@ -53,20 +53,35 @@ class ParDoBoundMultiTranslator
 
   @Override
   public void translate(ParDo.BoundMulti transform, 
TranslationContext context) {
-OldDoFn doFn = transform.getFn();
+DoFn doFn = transform.getNewFn();
+if 
(DoFnSignatures.getSignature(doFn.getClass()).stateDeclarations().size() > 0) {
+  throw new UnsupportedOperationException(
+  String.format(
+  "Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
+  DoFn.StateId.class.getSimpleName(),
+  doFn.getClass().getName(),
+  DoFn.class.getSimpleName(),
+  ApexRunner.class.getSimpleName()));
+}
+OldDoFn oldDoFn = transform.getFn();
 PCollectionTuple output = context.getOutput();
 PCollection input = context.getInput();
 List> sideInputs = transform.getSideInputs();
 Coder inputCoder = input.getCoder();
-WindowedValueCoder wvInputCoder = 
FullWindowedValueCoder.of(inputCoder,
-input.getWindowingStrategy().getWindowFn().windowCoder());
+WindowedValueCoder wvInputCoder =
+FullWindowedValueCoder.of(
+inputCoder, 
input.getWindowingStrategy().getWindowFn().windowCoder());
 
-ApexParDoOperator operator = new ApexParDoOperator<>(
-context.getPipelineOptions(),
-doFn, transform.getMainOutputTag(), 
transform.getSideOutputTags().getAll(),
-context.>getInput().getWindowingStrategy(), sideInputs, 
wvInputCoder,
-context.stateInternalsFactory()
-);
+ApexParDoOperator operator =
+new ApexParDoOperator<>(
+context.getPipelineOptions(),
+oldDoFn,
+transform.getMainOutputTag(),
+transform.getSideOutputTags().getAll(),
+context.>getInput().getWindowingStrategy(),
+sideInputs,
+wvInputCoder,
+context.stateInternalsFactory());
 
 

[29/50] incubator-beam git commit: This closes #1403

[kenn]

This closes #1403


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/70efa47c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/70efa47c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/70efa47c

Branch: refs/heads/gearpump-runner
Commit: 70efa47cf8b37dc1ad064b698ae49616c442bf19
Parents: 869d082 1b52070
Author: Davor Bonaci 
Authored: Mon Nov 21 14:32:39 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 14:32:39 2016 -0800

--
 sdks/java/microbenchmarks/README.md |  42 
 sdks/java/microbenchmarks/pom.xml   | 113 -
 .../coders/AvroCoderBenchmark.java  | 119 --
 .../coders/ByteArrayCoderBenchmark.java |  64 -
 .../coders/CoderBenchmarking.java   |  41 
 .../coders/StringUtf8CoderBenchmark.java|  70 --
 .../microbenchmarks/coders/package-info.java|  22 --
 .../transforms/DoFnInvokersBenchmark.java   | 232 ---
 .../transforms/package-info.java|  22 --
 sdks/java/pom.xml   |   1 -
 10 files changed, 726 deletions(-)
--

[25/50] incubator-beam git commit: This closes #1402

[kenn]

This closes #1402


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cb5d216b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cb5d216b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cb5d216b

Branch: refs/heads/gearpump-runner
Commit: cb5d216b69ec53f3522621c0acc75c5b364b767f
Parents: c6b86f6 30899db
Author: Davor Bonaci 
Authored: Mon Nov 21 13:33:12 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 21 13:33:12 2016 -0800

--
 README.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--

[18/50] incubator-beam git commit: Separate ParDoTest cases and make them more flexible

[kenn]

Separate ParDoTest cases and make them more flexible

A number of excessively rigid name tests preclude runner expansion
of ParDo. This change makes them into independent unit tests
for better signal and makes them more accurate to the intent - the
name should have the relevant information, but may have other
content.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e92a157a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e92a157a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e92a157a

Branch: refs/heads/gearpump-runner
Commit: e92a157aa41746df225379b71cc88695c8e2d93f
Parents: 3548ffb
Author: Kenneth Knowles 
Authored: Thu Nov 17 10:14:07 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../apache/beam/sdk/transforms/ParDoTest.java   | 55 ++--
 1 file changed, 27 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e92a157a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
index 26f5570..3c3e266 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
@@ -27,6 +27,7 @@ import static 
org.apache.beam.sdk.util.StringUtils.byteArrayToJsonString;
 import static org.apache.beam.sdk.util.StringUtils.jsonStringToByteArray;
 import static org.hamcrest.Matchers.allOf;
 import static org.hamcrest.Matchers.anyOf;
+import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.equalTo;
 import static 
org.hamcrest.collection.IsIterableContainingInAnyOrder.containsInAnyOrder;
 import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
@@ -826,39 +827,37 @@ public class ParDoTest implements Serializable {
   }
 
   @Test
-  public void testParDoGetName() {
+  public void testParDoOutputNameBasedOnDoFnWithTrimmedSuffix() {
 Pipeline p = TestPipeline.create();
+PCollection output = p.apply(Create.of(1)).apply(ParDo.of(new 
TestOldDoFn()));
+assertThat(output.getName(), containsString("ParDo(Test)"));
+  }
 
-PCollection input =
-p.apply(Create.of(Arrays.asList(3, -42, 666)))
-.setName("MyInput");
-
-{
-  PCollection output1 = input.apply(ParDo.of(new TestOldDoFn()));
-  assertEquals("ParDo(Test).out", output1.getName());
-}
-
-{
-  PCollection output2 = input.apply("MyParDo", ParDo.of(new 
TestOldDoFn()));
-  assertEquals("MyParDo.out", output2.getName());
-}
-
-{
-  PCollection output4 = input.apply("TestOldDoFn", ParDo.of(new 
TestOldDoFn()));
-  assertEquals("TestOldDoFn.out", output4.getName());
-}
+  @Test
+  public void testParDoOutputNameBasedOnLabel() {
+Pipeline p = TestPipeline.create();
+PCollection output =
+p.apply(Create.of(1)).apply("MyParDo", ParDo.of(new TestOldDoFn()));
+assertThat(output.getName(), containsString("MyParDo"));
+  }
 
-{
-  PCollection output5 = input.apply(ParDo.of(new 
StrangelyNamedDoer()));
-  assertEquals("ParDo(StrangelyNamedDoer).out",
-  output5.getName());
-}
+  @Test
+  public void testParDoOutputNameBasedDoFnWithoutMatchingSuffix() {
+Pipeline p = TestPipeline.create();
+PCollection output = p.apply(Create.of(1)).apply(ParDo.of(new 
StrangelyNamedDoer()));
+assertThat(output.getName(), containsString("ParDo(StrangelyNamedDoer)"));
+  }
 
-assertEquals("ParDo(Printing)", ParDo.of(new PrintingDoFn()).getName());
+  @Test
+  public void testParDoTransformNameBasedDoFnWithTrimmedSuffix() {
+assertThat(ParDo.of(new PrintingDoFn()).getName(), 
containsString("ParDo(Printing)"));
+  }
 
-assertEquals(
-"ParMultiDo(SideOutputDummy)",
-ParDo.of(new SideOutputDummyFn(null)).withOutputTags(null, 
null).getName());
+  @Test
+  public void testParDoMultiNameBasedDoFnWithTrimmerSuffix() {
+assertThat(
+ParDo.of(new SideOutputDummyFn(null)).withOutputTags(null, 
null).getName(),
+containsString("ParMultiDo(SideOutputDummy)"));
   }
 
   @Test

[39/50] incubator-beam git commit: [BEAM-952] Use ServiceLoader to register IOChannelFactories.

[kenn]

[BEAM-952] Use ServiceLoader to register IOChannelFactories.

This closes #1255


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e53d6d45
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e53d6d45
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e53d6d45

Branch: refs/heads/gearpump-runner
Commit: e53d6d45895aee6c61fd8060b20751875352a4ee
Parents: 249dbc0 fa417f9
Author: Luke Cwik 
Authored: Tue Nov 22 06:19:30 2016 -0800
Committer: Luke Cwik 
Committed: Tue Nov 22 06:19:30 2016 -0800

--
 .../beam/runners/dataflow/DataflowRunner.java   |   2 +-
 .../options/DataflowPipelineOptionsTest.java|   6 +-
 .../runners/dataflow/util/PackageUtilTest.java  |   2 +-
 .../sdk/options/PipelineOptionsFactory.java |  32 +
 .../apache/beam/sdk/runners/PipelineRunner.java |   2 +-
 .../apache/beam/sdk/testing/TestPipeline.java   |   2 +-
 .../beam/sdk/util/FileIOChannelFactory.java |  11 ++
 .../sdk/util/FileIOChannelFactoryRegistrar.java |  38 ++
 .../beam/sdk/util/GcsIOChannelFactory.java  |  10 +-
 .../sdk/util/GcsIOChannelFactoryRegistrar.java  |  38 ++
 .../sdk/util/IOChannelFactoryRegistrar.java |  48 +++
 .../apache/beam/sdk/util/IOChannelUtils.java| 134 ++-
 .../beam/sdk/util/common/ReflectHelpers.java|  29 
 .../java/org/apache/beam/sdk/io/AvroIOTest.java |   2 +-
 .../apache/beam/sdk/io/FileBasedSourceTest.java |   2 +-
 .../java/org/apache/beam/sdk/io/TextIOTest.java |   2 +-
 .../sdk/options/PipelineOptionsFactoryTest.java |  34 -
 .../util/FileIOChannelFactoryRegistrarTest.java |  44 ++
 .../beam/sdk/util/FileIOChannelFactoryTest.java |   2 +-
 .../util/GcsIOChannelFactoryRegistrarTest.java  |  44 ++
 .../beam/sdk/util/GcsIOChannelFactoryTest.java  |   2 +-
 .../beam/sdk/util/IOChannelUtilsTest.java   |  39 ++
 .../sdk/util/common/ReflectHelpersTest.java |  33 +
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java |   6 +-
 24 files changed, 478 insertions(+), 86 deletions(-)
--

[02/50] incubator-beam git commit: Replaces SideInputAccess with SideInputReader

[kenn]

Replaces SideInputAccess with SideInputReader

Makes WindowingInternals.sideInput take the side input window
instead of main input window.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/90a0d0e1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/90a0d0e1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/90a0d0e1

Branch: refs/heads/gearpump-runner
Commit: 90a0d0e13fa0332df805b79b1dc64860d9590217
Parents: 8243fcd
Author: Eugene Kirpichov 
Authored: Mon Nov 14 14:48:31 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Nov 17 13:18:36 2016 -0800

--
 .../operators/ApexGroupByKeyOperator.java   | 26 ++---
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  2 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |  2 +-
 .../runners/core/ReduceFnContextFactory.java| 27 +-
 .../beam/runners/core/ReduceFnRunner.java   |  5 ++-
 .../beam/runners/core/SideInputAccess.java  | 31 
 .../beam/runners/core/SimpleDoFnRunner.java | 32 +++-
 .../beam/runners/core/SimpleOldDoFnRunner.java  | 11 +++---
 .../core/WindowingInternalsAdapters.java| 21 ---
 .../beam/runners/core/ReduceFnTester.java   | 34 +
 .../GroupAlsoByWindowEvaluatorFactory.java  | 21 +++
 .../functions/FlinkDoFnFunction.java| 15 
 .../functions/FlinkProcessContextBase.java  |  9 +
 .../FlinkSingleOutputProcessContext.java|  1 -
 .../runners/spark/translation/DoFnFunction.java |  5 +--
 .../spark/translation/MultiDoFnFunction.java|  6 +--
 .../spark/translation/SparkProcessContext.java  | 39 
 .../apache/beam/sdk/transforms/DoFnTester.java  |  2 +-
 .../beam/sdk/util/WindowingInternals.java   |  4 +-
 19 files changed, 141 insertions(+), 152 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90a0d0e1/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 8fbfb03..eca4308 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -354,13 +354,26 @@ public class ApexGroupByKeyOperator implements 
Operator {
 }
 
 @Override
-public void outputWindowedValue(KV> output, Instant 
timestamp,
-Collection windows, PaneInfo pane) {
+public void outputWindowedValue(
+KV> output,
+Instant timestamp,
+Collection windows,
+PaneInfo pane) {
   if (traceTuples) {
 LOG.debug("\nemitting {} timestamp {}\n", output, timestamp);
   }
-  ApexGroupByKeyOperator.this.output.emit(ApexStreamTuple.DataTuple.of(
-  WindowedValue.of(output, timestamp, windows, pane)));
+  ApexGroupByKeyOperator.this.output.emit(
+  ApexStreamTuple.DataTuple.of(WindowedValue.of(output, timestamp, 
windows, pane)));
+}
+
+@Override
+public  void sideOutputWindowedValue(
+TupleTag tag,
+SideOutputT output,
+Instant timestamp,
+Collection windows,
+PaneInfo pane) {
+  throw new UnsupportedOperationException("GroupAlsoByWindow should 
not use side outputs");
 }
 
 @Override
@@ -379,8 +392,9 @@ public class ApexGroupByKeyOperator implements 
Operator {
 }
 
 @Override
-public  void writePCollectionViewData(TupleTag tag, 
Iterable> data,
-Coder elemCoder) throws IOException {
+public  void writePCollectionViewData(
+TupleTag tag, Iterable> data, Coder 
elemCoder)
+throws IOException {
   throw new RuntimeException("writePCollectionViewData() not available 
in Streaming mode.");
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90a0d0e1/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowViaWindowSetDoFn.java
index bcc52d3..8b10813 100644
--- 
a/runners/core-j

[48/50] incubator-beam git commit: Output Keyed Bundles in GroupAlsoByWindowEvaluator

[kenn]

Output Keyed Bundles in GroupAlsoByWindowEvaluator

This allows reuse of keys for downstream serialization.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1cec9702
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1cec9702
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1cec9702

Branch: refs/heads/gearpump-runner
Commit: 1cec9702e62b64252149645627d96889edfeb33e
Parents: b41789e
Author: Thomas Groh 
Authored: Tue Nov 22 14:51:39 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 15:05:17 2016 -0800

--
 .../main/java/org/apache/beam/runners/direct/DirectRunner.java  | 5 -
 .../beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java  | 4 +++-
 2 files changed, 7 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1cec9702/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index 0060e84..cb31947 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -31,6 +31,7 @@ import java.util.Map;
 import java.util.Set;
 import javax.annotation.Nullable;
 import org.apache.beam.runners.core.GBKIntoKeyedWorkItems;
+import org.apache.beam.runners.direct.DirectGroupByKey.DirectGroupAlsoByWindow;
 import org.apache.beam.runners.direct.DirectGroupByKey.DirectGroupByKeyOnly;
 import org.apache.beam.runners.direct.DirectRunner.DirectPipelineResult;
 import 
org.apache.beam.runners.direct.TestStreamEvaluatorFactory.DirectTestStreamFactory;
@@ -310,7 +311,9 @@ public class DirectRunner
 KeyedPValueTrackingVisitor keyedPValueVisitor =
 KeyedPValueTrackingVisitor.create(
 ImmutableSet.>of(
-GroupByKey.class, DirectGroupByKeyOnly.class));
+GBKIntoKeyedWorkItems.class,
+DirectGroupByKeyOnly.class,
+DirectGroupAlsoByWindow.class));
 pipeline.traverseTopologically(keyedPValueVisitor);
 
 DisplayDataValidator.validatePipeline(pipeline);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1cec9702/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
index b946e4d..36c742b 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
@@ -112,6 +112,7 @@ class GroupAlsoByWindowEvaluatorFactory implements 
TransformEvaluatorFactory {
 private @SuppressWarnings("unchecked") final WindowingStrategy
 windowingStrategy;
 
+private final StructuralKey structuralKey;
 private final Collection> outputBundles;
 private final ImmutableList.Builder>> 
unprocessedElements;
 private final AggregatorContainer.Mutator aggregatorChanges;
@@ -130,6 +131,7 @@ class GroupAlsoByWindowEvaluatorFactory implements 
TransformEvaluatorFactory {
   this.evaluationContext = evaluationContext;
   this.application = application;
 
+  structuralKey = inputBundle.getKey();
   stepContext = evaluationContext
   .getExecutionContext(application, inputBundle.getKey())
   .getOrCreateStepContext(
@@ -159,7 +161,7 @@ class GroupAlsoByWindowEvaluatorFactory implements 
TransformEvaluatorFactory {
   K key = workItem.key();
 
   UncommittedBundle>> bundle =
-  evaluationContext.createBundle(application.getOutput());
+  evaluationContext.createKeyedBundle(structuralKey, 
application.getOutput());
   outputBundles.add(bundle);
   CopyOnAccessInMemoryStateInternals stateInternals =
   (CopyOnAccessInMemoryStateInternals) stepContext.stateInternals();

[43/50] incubator-beam git commit: Reject stateful DoFn in FlinkRunner

[kenn]

Reject stateful DoFn in FlinkRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9c2d5da7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9c2d5da7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9c2d5da7

Branch: refs/heads/gearpump-runner
Commit: 9c2d5da7c659a2603d37c492ff44f4a9cda387fe
Parents: 7949b70
Author: Kenneth Knowles 
Authored: Tue Nov 15 21:33:28 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 10:52:39 2016 -0800

--
 runners/flink/runner/pom.xml|  1 +
 .../FlinkBatchTransformTranslators.java | 34 +---
 .../FlinkStreamingTransformTranslators.java | 25 +-
 3 files changed, 55 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9c2d5da7/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index adcb3de..c060c25 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -53,6 +53,7 @@
 
 
   
org.apache.beam.sdk.testing.RunnableOnService
+  
org.apache.beam.sdk.testing.UsesStatefulParDo
   none
   true
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9c2d5da7/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
index 935a9ac..474d4e3 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
@@ -24,6 +24,7 @@ import java.util.Collections;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import org.apache.beam.runners.flink.FlinkRunner;
 import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
 import org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction;
 import 
org.apache.beam.runners.flink.translation.functions.FlinkMergingNonShuffleReduceFunction;
@@ -46,6 +47,7 @@ import org.apache.beam.sdk.io.BoundedSource;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.transforms.Combine;
 import org.apache.beam.sdk.transforms.CombineFnBase;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.OldDoFn;
@@ -54,6 +56,7 @@ import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.join.RawUnionValue;
 import org.apache.beam.sdk.transforms.join.UnionCoder;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
@@ -487,11 +490,23 @@ class FlinkBatchTransformTranslators {
 @Override
 public void translateNode(
 ParDo.Bound transform,
+
 FlinkBatchTranslationContext context) {
+  DoFn doFn = transform.getNewFn();
+  if 
(DoFnSignatures.getSignature(doFn.getClass()).stateDeclarations().size() > 0) {
+throw new UnsupportedOperationException(
+String.format(
+"Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
+DoFn.StateId.class.getSimpleName(),
+doFn.getClass().getName(),
+DoFn.class.getSimpleName(),
+FlinkRunner.class.getSimpleName()));
+  }
+
   DataSet> inputDataSet =
   context.getInputDataSet(context.getInput(transform));
 
-  final OldDoFn doFn = transform.getFn();
+  final OldDoFn oldDoFn = transform.getFn();
 
   TypeInformation> typeInformation =
   context.getTypeInfo(context.getOutput(transform));
@@ -507,7 +522,7 @@ class FlinkBatchTransformTranslators {
 
   FlinkDoFnFunction doFnWrapper =
   new FlinkDoFnFunction<>(
-  doFn,
+  oldDoFn,
   context.getOutput(transform).getWindowingStrategy(),
   sideInputStrategies,
   context.getPipelineOptions());
@@ -533,10 +548,21 @@ class Fli

[12/50] incubator-beam git commit: Restore StateContexts.windowOnly for temporary compatibility

[kenn]

Restore StateContexts.windowOnly for temporary compatibility


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/73692b80
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/73692b80
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/73692b80

Branch: refs/heads/gearpump-runner
Commit: 73692b808aa4e0d85c6e20eb19a1bf4a4355a5c1
Parents: 2a7169b
Author: Kenneth Knowles 
Authored: Fri Nov 18 14:40:59 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 14:59:41 2016 -0800

--
 .../beam/sdk/util/state/StateContexts.java  | 41 +++-
 1 file changed, 32 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/73692b80/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
index 858d6fe..1d6e949 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
@@ -42,22 +42,45 @@ public class StateContexts {
 @Override
 public BoundedWindow window() {
   throw new IllegalArgumentException("cannot call window() in a null 
context");
-}};
+}
+  };
 
-  /**
-   * Returns a fake {@link StateContext}.
-   */
+  /** Returns a fake {@link StateContext}. */
   @SuppressWarnings("unchecked")
   public static  StateContext nullContext() {
 return (StateContext) NULL_CONTEXT;
   }
 
   /**
-   * Deprecated, do not use.
-   *
-   * This exists only for temporary compatibility with Dataflow worker and 
should be deleted
-   * once a worker image is released that uses runners-core build after
-   * https://github.com/apache/incubator-beam/pull/1353.
+   * @deprecated This exists only for temporary compatibility with Dataflow 
worker and should be
+   * deleted once a worker image is released that uses runners-core build 
after
+   * https://github.com/apache/incubator-beam/pull/1353.
+   */
+  @Deprecated
+  public static  StateContext windowOnly(final W 
window) {
+return new StateContext() {
+  @Override
+  public PipelineOptions getPipelineOptions() {
+throw new IllegalArgumentException(
+"cannot call getPipelineOptions() in a window only context");
+  }
+
+  @Override
+  public  T sideInput(PCollectionView view) {
+throw new IllegalArgumentException("cannot call sideInput() in a 
window only context");
+  }
+
+  @Override
+  public W window() {
+return window;
+  }
+};
+  }
+
+  /**
+   * @deprecated This exists only for temporary compatibility with Dataflow 
worker and should be
+   * deleted once a worker image is released that uses runners-core build 
after
+   * https://github.com/apache/incubator-beam/pull/1353.
*/
   @Deprecated
   public static  StateContext createFromComponents(

[19/50] incubator-beam git commit: Delegate getAggregators() in various DoFn adapters

[kenn]

Delegate getAggregators() in various DoFn adapters


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b0d46c2d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b0d46c2d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b0d46c2d

Branch: refs/heads/gearpump-runner
Commit: b0d46c2deb4318f8d0e55eeeb20e1d11ceadd218
Parents: 6fa8057
Author: Kenneth Knowles 
Authored: Thu Nov 17 15:50:17 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../main/java/org/apache/beam/sdk/transforms/DoFn.java   |  7 ++-
 .../org/apache/beam/sdk/transforms/DoFnAdapters.java |  6 ++
 .../java/org/apache/beam/sdk/transforms/OldDoFn.java |  7 ++-
 .../java/org/apache/beam/sdk/transforms/OldDoFnTest.java | 11 +++
 4 files changed, 25 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b0d46c2d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
index 9978ef4..221d942 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
@@ -28,6 +28,8 @@ import java.lang.annotation.ElementType;
 import java.lang.annotation.Retention;
 import java.lang.annotation.RetentionPolicy;
 import java.lang.annotation.Target;
+import java.util.Collection;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.Map;
 import javax.annotation.Nullable;
@@ -236,7 +238,6 @@ public abstract class DoFn implements 
Serializable, HasDisplayD
 
   aggregator.setDelegate(delegate);
 }
-
   }
 
   /**
@@ -298,6 +299,10 @@ public abstract class DoFn implements 
Serializable, HasDisplayD
 
   protected Map> aggregators = new 
HashMap<>();
 
+  Collection> getAggregators() {
+return Collections.>unmodifiableCollection(aggregators.values());
+  }
+
   /**
* Protects aggregators from being created after initialization.
*/

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b0d46c2d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
index a3466bb..1a74ae7 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
@@ -18,6 +18,7 @@
 package org.apache.beam.sdk.transforms;
 
 import java.io.IOException;
+import java.util.Collection;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
 import org.apache.beam.sdk.transforms.DoFn.Context;
@@ -244,6 +245,11 @@ public class DoFnAdapters {
 }
 
 @Override
+Collection> getAggregators() {
+  return fn.getAggregators();
+}
+
+@Override
 public Duration getAllowedTimestampSkew() {
   return fn.getAllowedTimestampSkew();
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b0d46c2d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
index f16e0b3..9bf9003 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
@@ -676,6 +676,11 @@ public abstract class OldDoFn implements 
Serializable, HasDispl
 }
 
 @Override
+Collection> getAggregators() {
+  return OldDoFn.this.getAggregators();
+}
+
+@Override
 protected TypeDescriptor getOutputTypeDescriptor() {
   return OldDoFn.this.getOutputTypeDescriptor();
 }
@@ -683,7 +688,7 @@ public abstract class OldDoFn implements 
Serializable, HasDispl
 
   /**
* A {@link ProcessContext} for an {@link OldDoFn} that implements
-   * {@link OldDoFn.RequiresWindowAcccess}, via a context for a proper {@link 
DoFn}.
+   * {@link RequiresWindowAccess}, via a context for a proper {@link DoFn}.
*/
   private class AdaptedRequiresWindowAccessProcessContext extends 
AdaptedProcessContext {
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b0d46c2d/sdks/java/core/src/test/java/or

[49/50] incubator-beam git commit: This closes #1422

[kenn]

This closes #1422


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/dd85cad0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/dd85cad0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/dd85cad0

Branch: refs/heads/gearpump-runner
Commit: dd85cad0284d266fc569ecc6d98653bf8aa5d031
Parents: b41789e 1cec970
Author: Kenneth Knowles 
Authored: Tue Nov 22 15:05:17 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 15:05:17 2016 -0800

--
 .../main/java/org/apache/beam/runners/direct/DirectRunner.java  | 5 -
 .../beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java  | 4 +++-
 2 files changed, 7 insertions(+), 2 deletions(-)
--

[17/50] incubator-beam git commit: Do not override type descriptor in WithTimestamps

[kenn]

Do not override type descriptor in WithTimestamps


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6fa8057f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6fa8057f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6fa8057f

Branch: refs/heads/gearpump-runner
Commit: 6fa8057fe1db97615872915dcd81b2bca9b44e63
Parents: e92a157
Author: Kenneth Knowles 
Authored: Thu Nov 17 11:07:32 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../java/org/apache/beam/sdk/transforms/WithTimestamps.java | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6fa8057f/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
index 00ac8e4..64e7c45 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
@@ -99,9 +99,8 @@ public class WithTimestamps extends 
PTransform, PCollection
 
   @Override
   public PCollection apply(PCollection input) {
-return input
-.apply("AddTimestamps", ParDo.of(new AddTimestampsDoFn(fn, 
allowedTimestampSkew)))
-.setTypeDescriptorInternal(input.getTypeDescriptor());
+return input.apply(
+"AddTimestamps", ParDo.of(new AddTimestampsDoFn(fn, 
allowedTimestampSkew)));
   }
 
   private static class AddTimestampsDoFn extends DoFn {

[46/50] incubator-beam git commit: This closes #1417

[kenn]

This closes #1417


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b41789e9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b41789e9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b41789e9

Branch: refs/heads/gearpump-runner
Commit: b41789e9c5ac8691243c796968b00a65cc11dd39
Parents: 6ec45f7 e6870a6
Author: Thomas Groh 
Authored: Tue Nov 22 12:27:41 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 22 12:27:41 2016 -0800

--
 .../beam/runners/direct/TransformExecutor.java  |  5 +-
 .../beam/sdk/metrics/MetricsEnvironment.java| 60 +++-
 .../sdk/metrics/MetricsEnvironmentTest.java |  8 +--
 .../apache/beam/sdk/metrics/MetricsTest.java|  6 +-
 4 files changed, 56 insertions(+), 23 deletions(-)
--

[GitHub] incubator-beam pull request #1425: Add TransformHierarchyTest

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1425


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: This closes #1425

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master dd85cad02 -> 3ae9425b3


This closes #1425


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3ae9425b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3ae9425b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3ae9425b

Branch: refs/heads/master
Commit: 3ae9425b3a36e34890980647952c61cedbd91ff3
Parents: dd85cad 73d0af9
Author: Kenneth Knowles 
Authored: Tue Nov 22 22:24:37 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 22:24:37 2016 -0800

--
 .../sdk/runners/TransformHierarchyTest.java | 142 +++
 1 file changed, 142 insertions(+)
--

[2/2] incubator-beam git commit: Add TransformHierarchyTest

[kenn]

Add TransformHierarchyTest

This tests basic features of TransformHierarchy


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/73d0af91
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/73d0af91
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/73d0af91

Branch: refs/heads/master
Commit: 73d0af9151340d85df1f720e88366f8a463b44bc
Parents: dd85cad
Author: Thomas Groh 
Authored: Tue Nov 22 16:14:29 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 22:24:37 2016 -0800

--
 .../sdk/runners/TransformHierarchyTest.java | 142 +++
 1 file changed, 142 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/73d0af91/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
new file mode 100644
index 000..c28f23e
--- /dev/null
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.runners;
+
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.junit.Assert.assertThat;
+
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.beam.sdk.Pipeline.PipelineVisitor;
+import org.apache.beam.sdk.io.CountingSource;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PValue;
+import org.hamcrest.Matchers;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for {@link TransformHierarchy}.
+ */
+@RunWith(JUnit4.class)
+public class TransformHierarchyTest {
+  @Rule public ExpectedException thrown = ExpectedException.none();
+  private TransformHierarchy hierarchy;
+  private TestPipeline pipeline;
+
+  @Before
+  public void setup() {
+hierarchy = new TransformHierarchy();
+pipeline = TestPipeline.create();
+  }
+
+  @Test
+  public void getCurrentNoPushReturnsRoot() {
+assertThat(hierarchy.getCurrent().isRootNode(), is(true));
+  }
+
+  @Test
+  public void popWithoutPushThrows() {
+thrown.expect(IllegalStateException.class);
+hierarchy.popNode();
+  }
+
+  @Test
+  public void pushThenPopSucceeds() {
+TransformTreeNode root = hierarchy.getCurrent();
+TransformTreeNode node =
+new TransformTreeNode(hierarchy.getCurrent(), Create.of(1), "Create", 
PBegin.in(pipeline));
+hierarchy.pushNode(node);
+assertThat(hierarchy.getCurrent(), equalTo(node));
+hierarchy.popNode();
+assertThat(hierarchy.getCurrent(), equalTo(root));
+  }
+
+  @Test
+  public void visitVisitsAllPushed() {
+TransformTreeNode root = hierarchy.getCurrent();
+Create.Values create = Create.of(1);
+PCollection created = pipeline.apply(create);
+PBegin begin = PBegin.in(pipeline);
+
+TransformTreeNode compositeNode =
+new TransformTreeNode(root, create, "Create", begin);
+root.addComposite(compositeNode);
+TransformTreeNode primitiveNode =
+new TransformTreeNode(
+compositeNode, Read.from(CountingSource.upTo(1L)), "Create/Read", 
begin);
+compositeNode.addComposite(primitiveNode);
+
+TransformTreeNode otherPrimitive =
+new TransformTreeNode(
+root, MapElements.via(new SimpleFunction()

[jira] [Commented] (BEAM-1025) User guide - "How to create Beam IO Transforms"

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15689087#comment-15689087
 ] 

Jean-Baptiste Onofré commented on BEAM-1025:


I will be happy to help there (as I did some IOs ;)).

> User guide - "How to create Beam IO Transforms"
> ---
>
> Key: BEAM-1025
> URL: https://issues.apache.org/jira/browse/BEAM-1025
> Project: Beam
>  Issue Type: Task
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> Beam has javadocs for how to create a read or write transform, but no 
> friendly user guide on how to get started using BoundedSource/BoundedReader.
> This should cover:
> * background on beam's source/sink API design 
> * design patterns
> * evaluating different data sources (eg, what are the properties of a pub sub 
> system that affect how you should write your UnboundedSource? What is the 
> best design for reading from a NoSql style source?)
> * testing - how to write unit, integration (and once we have them, 
> performance tests)
> * public API recommendations
> This is related, but not strictly overlapping with: 
> https://issues.apache.org/jira/browse/BEAM-193
> - the Dataflow SDK documentation for "Custom Sources and Sinks"  contains 
> some info about writing Sources/Sinks, but it is somewhat out of date, and 
> doesn't reflect the things we've learned recently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1042) Clean build fails on Windows in Starter archetype

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15689081#comment-15689081
 ] 

Jean-Baptiste Onofré commented on BEAM-1042:


Hi Thomas,

by the way, we have a PR opened about building on Windows: 
https://github.com/apache/incubator-beam/pull/496

I think this PR is not required anymore but we might have some overlap with 
your PR.

> Clean build fails on Windows in Starter archetype
> -
>
> Key: BEAM-1042
> URL: https://issues.apache.org/jira/browse/BEAM-1042
> Project: Beam
>  Issue Type: Bug
>Reporter: Thomas Weise
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1427: Break dependency from util.common -> util

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1427

Break dependency from util.common -> util

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @lukecwik 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam ReflectHelpers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1427.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1427


commit ab3a2f8d779abc80e7b2570e0c7cddcd9b7e2400
Author: Kenneth Knowles 
Date:   2016-11-23T06:16:29Z

Break dependency from util.common -> util




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam-site git commit: This closes #81

[davor]

This closes #81


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/946ed001
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/946ed001
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/946ed001

Branch: refs/heads/asf-site
Commit: 946ed001073e05282c83e8c8960d20719682908f
Parents: 51e0afe 6a45350
Author: Davor Bonaci 
Authored: Tue Nov 22 22:11:49 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Nov 22 22:11:49 2016 -0800

--
 .../pipelines/create-your-pipeline/index.html   | 174 ++-
 .../pipelines/design-your-pipeline/index.html   | 117 +++-
 .../pipelines/test-your-pipeline/index.html | 289 ++-
 content/images/design-your-pipeline-flatten.png | Bin 0 -> 47858 bytes
 content/images/design-your-pipeline-join.png| Bin 0 -> 41878 bytes
 content/images/design-your-pipeline-linear.png  | Bin 0 -> 15218 bytes
 ...sign-your-pipeline-multiple-pcollections.png | Bin 0 -> 39095 bytes
 .../design-your-pipeline-side-outputs.png   | Bin 0 -> 36451 bytes
 .../pipelines/create-your-pipeline.md   | 155 +-
 .../pipelines/design-your-pipeline.md   | 104 ++-
 .../pipelines/test-your-pipeline.md | 266 -
 src/images/design-your-pipeline-flatten.png | Bin 0 -> 47858 bytes
 src/images/design-your-pipeline-join.png| Bin 0 -> 41878 bytes
 src/images/design-your-pipeline-linear.png  | Bin 0 -> 15218 bytes
 ...sign-your-pipeline-multiple-pcollections.png | Bin 0 -> 39095 bytes
 .../design-your-pipeline-side-outputs.png   | Bin 0 -> 36451 bytes
 16 files changed, 1097 insertions(+), 8 deletions(-)
--

[2/3] incubator-beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/6a453509
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/6a453509
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/6a453509

Branch: refs/heads/asf-site
Commit: 6a45350997258dbae8bf2ffa6b712a5f9bff7130
Parents: 5b3bda6
Author: Davor Bonaci 
Authored: Tue Nov 22 22:11:49 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Nov 22 22:11:49 2016 -0800

--
 .../pipelines/create-your-pipeline/index.html   | 174 ++-
 .../pipelines/design-your-pipeline/index.html   | 117 +++-
 .../pipelines/test-your-pipeline/index.html | 289 ++-
 content/images/design-your-pipeline-flatten.png | Bin 0 -> 47858 bytes
 content/images/design-your-pipeline-join.png| Bin 0 -> 41878 bytes
 content/images/design-your-pipeline-linear.png  | Bin 0 -> 15218 bytes
 ...sign-your-pipeline-multiple-pcollections.png | Bin 0 -> 39095 bytes
 .../design-your-pipeline-side-outputs.png   | Bin 0 -> 36451 bytes
 8 files changed, 575 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/6a453509/content/documentation/pipelines/create-your-pipeline/index.html
--
diff --git a/content/documentation/pipelines/create-your-pipeline/index.html 
b/content/documentation/pipelines/create-your-pipeline/index.html
index c3c2182..83e43f9 100644
--- a/content/documentation/pipelines/create-your-pipeline/index.html
+++ b/content/documentation/pipelines/create-your-pipeline/index.html
@@ -143,10 +143,182 @@
   
 Create Your Pipeline
 
+
+  Creating Your Pipeline 
Object
+  Configuring Pipeline Options 
   
+  Setting 
PipelineOptions from Command-Line Arguments
+  Creating Custom Options
+
+  
+
+  
+  Reading Data Into Your 
Pipeline
+  Applying 
Transforms to Process Pipeline Data
+  Writing or 
Outputting Your Final Pipeline Data
+  Running Your Pipeline
+  What’s next
+
+
+Your Beam program expresses a data processing pipeline, from start to 
finish. This section explains the mechanics of using the classes in the Beam 
SDKs to build a pipeline. To construct a pipeline using the classes in the Beam 
SDKs, your program will need to perform the following general steps:
+
+
+  Create a Pipeline object.
+  Use a Read or Create transform to 
create one or more PCollections for your 
pipeline data.
+  Apply transforms to each PCollection. Transforms can change, filter, 
group, analyze, or otherwise process the elements in a PCollection. Each transform creates a new 
output PCollection, to which you can 
apply additional transforms until processing is complete.
+  Write or otherwise output the final, transformed PCollections.
+  Run the pipeline.
+
+
+Creating Your Pipeline Object
+
+A Beam program often starts by creating a Pipeline object.
+
+In the Beam SDKs, each pipeline is represented by an explicit object of 
type Pipeline. Each Pipeline object is an independent entity that 
encapsulates both the data the pipeline operates over and the transforms that 
get applied to that data.
+
+To create a pipeline, declare a Pipeline object, and pass it some 
configuration options, which are explained in a section below. You pass the 
configuration options by creating an object of type PipelineOptions, which you can build by using 
the static method PipelineOptionsFactory.create().
+
+// Start by defining the options for 
the pipeline.
+PipelineOptions options = PipelineOptionsFactory.create();
+
+// Then create the pipeline.
+Pipeline p = Pipeline.create(options);
+
+
+
+Configuring Pipeline Options
+
+Use the pipeline options to configure different aspects of your pipeline, 
such as the pipeline runner that will execute your pipeline and any 
runner-specific configuration required by the chosen runner. Your pipeline 
options will potentially include information such as your project ID or a 
location for storing files.
+
+When you run the pipeline on a runner of your choice, a copy of the 
PipelineOptions will be available to your code. For example, you can read 
PipelineOptions from a DoFn’s Context.
+
+Setting 
PipelineOptions from Command-Line Arguments
+
+While you can configure your pipeline by creating a PipelineOptions object and setting the fields 
directly, the Beam SDKs include a command-line parser that you can use to set 
fields in PipelineOptions using 
command-line arguments.
+
+To read options from the command-line, construct your PipelineOptions object as demonstrated in the 
following example code:
+
+MyOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().create();
+
+
+
+This interprets co

[1/3] incubator-beam-site git commit: Update documentation/pipelines/design-your-pipeline.md. Add 5 images for the doc.

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 51e0afeb0 -> 946ed0010


Update documentation/pipelines/design-your-pipeline.md. Add 5 images for the 
doc.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/5b3bda65
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/5b3bda65
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/5b3bda65

Branch: refs/heads/asf-site
Commit: 5b3bda6590361c2a7f8923d46780dbc72499440b
Parents: 51e0afe
Author: Hadar Hod 
Authored: Mon Nov 7 15:58:49 2016 -0800
Committer: Davor Bonaci 
Committed: Tue Nov 22 22:10:51 2016 -0800

--
 .../pipelines/create-your-pipeline.md   | 155 ++-
 .../pipelines/design-your-pipeline.md   | 104 +++-
 .../pipelines/test-your-pipeline.md | 266 ++-
 src/images/design-your-pipeline-flatten.png | Bin 0 -> 47858 bytes
 src/images/design-your-pipeline-join.png| Bin 0 -> 41878 bytes
 src/images/design-your-pipeline-linear.png  | Bin 0 -> 15218 bytes
 ...sign-your-pipeline-multiple-pcollections.png | Bin 0 -> 39095 bytes
 .../design-your-pipeline-side-outputs.png   | Bin 0 -> 36451 bytes
 8 files changed, 522 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5b3bda65/src/documentation/pipelines/create-your-pipeline.md
--
diff --git a/src/documentation/pipelines/create-your-pipeline.md 
b/src/documentation/pipelines/create-your-pipeline.md
index b99628f..05b1594 100644
--- a/src/documentation/pipelines/create-your-pipeline.md
+++ b/src/documentation/pipelines/create-your-pipeline.md
@@ -5,4 +5,157 @@ permalink: /documentation/pipelines/create-your-pipeline/
 ---
 # Create Your Pipeline
 
-> **Note:** There is an open JIRA issue to create this guide 
([BEAM-901](https://issues.apache.org/jira/browse/BEAM-901)).
+* TOC
+{:toc}
+
+Your Beam program expresses a data processing pipeline, from start to finish. 
This section explains the mechanics of using the classes in the Beam SDKs to 
build a pipeline. To construct a pipeline using the classes in the Beam SDKs, 
your program will need to perform the following general steps:
+
+*   Create a `Pipeline` object.
+*   Use a **Read** or **Create** transform to create one or more 
`PCollection`s for your pipeline data.
+*   Apply **transforms** to each `PCollection`. Transforms can change, filter, 
group, analyze, or otherwise process the elements in a `PCollection`. Each 
transform creates a new output `PCollection`, to which you can apply additional 
transforms until processing is complete.
+*   **Write** or otherwise output the final, transformed `PCollection`s.
+*   **Run** the pipeline.
+
+## Creating Your Pipeline Object
+
+A Beam program often starts by creating a `Pipeline` object.
+
+In the Beam SDKs, each pipeline is represented by an explicit object of type 
`Pipeline`. Each `Pipeline` object is an independent entity that encapsulates 
both the data the pipeline operates over and the transforms that get applied to 
that data.
+
+To create a pipeline, declare a `Pipeline` object, and pass it some 
configuration options, which are explained in a section below. You pass the 
configuration options by creating an object of type `PipelineOptions`, which 
you can build by using the static method `PipelineOptionsFactory.create()`.
+
+```java
+// Start by defining the options for the pipeline.
+PipelineOptions options = PipelineOptionsFactory.create();
+
+// Then create the pipeline.
+Pipeline p = Pipeline.create(options);
+```
+
+### Configuring Pipeline Options
+
+Use the pipeline options to configure different aspects of your pipeline, such 
as the pipeline runner that will execute your pipeline and any runner-specific 
configuration required by the chosen runner. Your pipeline options will 
potentially include information such as your project ID or a location for 
storing files. 
+
+When you run the pipeline on a runner of your choice, a copy of the 
PipelineOptions will be available to your code. For example, you can read 
PipelineOptions from a DoFn's Context.
+
+ Setting PipelineOptions from Command-Line Arguments
+
+While you can configure your pipeline by creating a `PipelineOptions` object 
and setting the fields directly, the Beam SDKs include a command-line parser 
that you can use to set fields in `PipelineOptions` using command-line 
arguments.
+
+To read options from the command-line, construct your `PipelineOptions` object 
as demonstrated in the following example code:
+
+```java
+MyOptions options = 
PipelineOptionsFactory.fromArgs(args).withValidation().create();
+```
+
+This interprets command-line arguments that 

[GitHub] incubator-beam pull request #1426: FileBasedSource: replace SeekableByteChan...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1426

FileBasedSource: replace SeekableByteChannel with open with 
startingPosition.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam io-channel-open-channel

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1426.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1426


commit 714a81399fa61c1f8f4795af9da1bd23d4ce294b
Author: Pei He 
Date:   2016-11-23T05:04:36Z

FileBasedSource: replace SeekableByteChannel with open with 
startingPosition.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1043) Dependency problems found in runner-apex

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-1043:
---
Affects Version/s: (was: 0.3.0-incubating)
   0.4.0-incubating

> Dependency problems found in runner-apex
> 
>
> Key: BEAM-1043
> URL: https://issues.apache.org/jira/browse/BEAM-1043
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Affects Versions: 0.4.0-incubating
>Reporter: Manu Zhang
>Assignee: Thomas Weise
>
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
> beam-runners-apex ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]commons-io:commons-io:jar:2.4:compile
> [WARNING]com.datatorrent:netlet:jar:1.3.0:compile
> [WARNING]org.apache.hadoop:hadoop-common:jar:2.6.0:compile



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1043) Dependency problems found in runner-apex

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688545#comment-15688545
 ] 

Thomas Weise commented on BEAM-1043:


https://github.com/apache/incubator-beam/pull/1418


> Dependency problems found in runner-apex
> 
>
> Key: BEAM-1043
> URL: https://issues.apache.org/jira/browse/BEAM-1043
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Affects Versions: 0.4.0-incubating
>Reporter: Manu Zhang
>Assignee: Thomas Weise
>
> [INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
> beam-runners-apex ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]commons-io:commons-io:jar:2.4:compile
> [WARNING]com.datatorrent:netlet:jar:1.3.0:compile
> [WARNING]org.apache.hadoop:hadoop-common:jar:2.6.0:compile



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1418: Update transitive dependencies for Apex 3...

[tweise]

GitHub user tweise reopened a pull request:

https://github.com/apache/incubator-beam/pull/1418

Update transitive dependencies for Apex 3.5.0 snapshot version.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
R @kennknowles 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tweise/incubator-beam apex-dependencies

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1418.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1418


commit 12ce2518c9339d8b6fac13b905f652be417af104
Author: Thomas Weise 
Date:   2016-11-22T19:38:00Z

Update transitive dependencies for Apex 3.5.0 snapshot version.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1418: Update transitive dependencies for Apex 3...

[tweise]

Github user tweise closed the pull request at:

https://github.com/apache/incubator-beam/pull/1418


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1043) Dependency problems found in runner-apex

[Manu Zhang (JIRA)]

Manu Zhang created BEAM-1043:


 Summary: Dependency problems found in runner-apex
 Key: BEAM-1043
 URL: https://issues.apache.org/jira/browse/BEAM-1043
 Project: Beam
  Issue Type: Bug
  Components: runner-apex
Affects Versions: 0.3.0-incubating
Reporter: Manu Zhang
Assignee: Thomas Weise


[INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
beam-runners-apex ---
[WARNING] Used undeclared dependencies found:
[WARNING]commons-io:commons-io:jar:2.4:compile
[WARNING]com.datatorrent:netlet:jar:1.3.0:compile
[WARNING]org.apache.hadoop:hadoop-common:jar:2.6.0:compile




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1042) Clean build fails on Windows in Starter archetype

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688466#comment-15688466
 ] 

Thomas Weise commented on BEAM-1042:


{code}
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test 
(default-integration-test) on project beam-sdks-java-maven-archetypes-starter:
Archetype IT 'basic' failed: Cannot run additions goals.
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException:
Archetype IT 'basic' failed: Cannot run additions goals.
at 
org.apache.maven.archetype.mojos.IntegrationTestMojo.execute(IntegrationTestMojo.java:269)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
{code}

> Clean build fails on Windows in Starter archetype
> -
>
> Key: BEAM-1042
> URL: https://issues.apache.org/jira/browse/BEAM-1042
> Project: Beam
>  Issue Type: Bug
>Reporter: Thomas Weise
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1042) Clean build fails on Windows in Starter archetype

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688459#comment-15688459
 ] 

Thomas Weise commented on BEAM-1042:


mvn clean install -DskipTests

{code}
[INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC .. SUCCESS [  1.621 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes  SUCCESS [  0.068 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter FAILURE [  
2.788 s]
[INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes :: Examples SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Join library SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Sorter  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Java 8 Tests  SKIPPED
[INFO] Apache Beam :: Runners :: Flink  SKIPPED
[INFO] Apache Beam :: Runners :: Flink :: Core  SKIPPED
[INFO] Apache Beam :: Runners :: Flink :: Examples  SKIPPED
[INFO] Apache Beam :: Runners :: Spark  SKIPPED
[INFO] Apache Beam :: Runners :: Apex . SKIPPED
[INFO] Apache Beam :: Examples  SKIPPED
[INFO] Apache Beam :: Examples :: Java  SKIPPED
[INFO] Apache Beam :: Examples :: Java 8 .. SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 01:26 min
[INFO] Finished at: 2016-11-22T16:42:51-08:00
[INFO] Final Memory: 130M/1409M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-archetype-plugin:2.4:integration-test 
(default-integration-test) on project beam-sdks-java-maven-archetypes-starter:
[ERROR] Archetype IT 'basic' failed: Cannot run additions goals.
[ERROR] -> [Help 1]
{code}


> Clean build fails on Windows in Starter archetype
> -
>
> Key: BEAM-1042
> URL: https://issues.apache.org/jira/browse/BEAM-1042
> Project: Beam
>  Issue Type: Bug
>Reporter: Thomas Weise
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1042) Clean build fails on Windows in Starter archetype

[Thomas Weise (JIRA)]

Thomas Weise created BEAM-1042:
--

 Summary: Clean build fails on Windows in Starter archetype
 Key: BEAM-1042
 URL: https://issues.apache.org/jira/browse/BEAM-1042
 Project: Beam
  Issue Type: Bug
Reporter: Thomas Weise






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1425: Add TransformHierarchyTest

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1425

Add TransformHierarchyTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This tests basic features of TransformHierarchy

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam transform_hierarchy_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1425


commit 7d72601b9be356c35e5dd25b86b0857ad4ce554a
Author: Thomas Groh 
Date:   2016-11-23T00:14:29Z

Add TransformHierarchyTest

This tests basic features of TransformHierarchy




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1424: Add input and output types to TransformRe...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1424

Add input and output types to TransformResult

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This would likely have caught some hard-to-diagnose (for me, new to the 
runner) type safety errors during the development of 
`StatefulParDoEvaluatorFactory`, so adding it should hopefully catch similar 
bugs in the future. As we discussed `InputT` is more useful and reasonable than 
`OutputT` but there are a number of places where they each line up and might 
add a little something.

It adds some boilerplate, to be sure, and is still mostly type unsafe since 
there are lots of methods with generics only on the return type. Opening this 
for consideration / discussion.

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
typed-TransformResult

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1424


commit 4e8a1157a88d8b7ecd64e75c46b0cd6fbbba03ee
Author: Kenneth Knowles 
Date:   2016-11-23T00:01:45Z

Add input and output types to TransformResult

This would likely have caught some hard-to-diagnose type safety errors
during the development of StatefulParDoEvaluatorFactory, so adding it
should hopefully catch similar bugs in the future.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1041) IOChannelFactory: supports for recursively remove().

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-1041:
-
Summary: IOChannelFactory: supports for recursively remove().  (was: 
IOChannelFactory: supports for recursively remove.)

> IOChannelFactory: supports for recursively remove().
> 
>
> Key: BEAM-1041
> URL: https://issues.apache.org/jira/browse/BEAM-1041
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>
> 1. FileIOChannelFactory:
> Maybe using deleteDirectory() as suggested in here:
> http://stackoverflow.com/questions/779519/delete-directories-recursively-in-java
> 2. GcsIOChannelFactory:
> Since gcs doesn't have directories, remove recursively means remove all files 
> whose prefixes match.
> However, there are two issues to consider:
> First. remove("gs://mybucket/directory-to-remove")
> a. it should remove "gs://mybucket/directory-to-remove/my-file".
> b. it should not remove "gs://mybucket/directory-to-remove-my-file".
> Second, there should be a way to skip the prefix match for 
> remove(collection-of-a-thousand-files), perhaps by adding a recursive boolean 
> parameter to remove().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1041) IOChannelFactory: supports for recursively remove.

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-1041:
-
Assignee: (was: Davor Bonaci)

> IOChannelFactory: supports for recursively remove.
> --
>
> Key: BEAM-1041
> URL: https://issues.apache.org/jira/browse/BEAM-1041
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>
> 1. FileIOChannelFactory:
> Maybe using deleteDirectory() as suggested in here:
> http://stackoverflow.com/questions/779519/delete-directories-recursively-in-java
> 2. GcsIOChannelFactory:
> Since gcs doesn't have directories, remove recursively means remove all files 
> whose prefixes match.
> However, there are two issues to consider:
> First. remove("gs://mybucket/directory-to-remove")
> a. it should remove "gs://mybucket/directory-to-remove/my-file".
> b. it should not remove "gs://mybucket/directory-to-remove-my-file".
> Second, there should be a way to skip the prefix match for 
> remove(collection-of-a-thousand-files), perhaps by adding a recursive boolean 
> parameter to remove().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1041) IOChannelFactory: supports for recursively remove.

[Pei He (JIRA)]

Pei He created BEAM-1041:


 Summary: IOChannelFactory: supports for recursively remove.
 Key: BEAM-1041
 URL: https://issues.apache.org/jira/browse/BEAM-1041
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Pei He
Assignee: Davor Bonaci


1. FileIOChannelFactory:
Maybe using deleteDirectory() as suggested in here:
http://stackoverflow.com/questions/779519/delete-directories-recursively-in-java

2. GcsIOChannelFactory:
Since gcs doesn't have directories, remove recursively means remove all files 
whose prefixes match.

However, there are two issues to consider:
First. remove("gs://mybucket/directory-to-remove")
a. it should remove "gs://mybucket/directory-to-remove/my-file".
b. it should not remove "gs://mybucket/directory-to-remove-my-file".

Second, there should be a way to skip the prefix match for 
remove(collection-of-a-thousand-files), perhaps by adding a recursive boolean 
parameter to remove().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Remove tests for merge

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 3b5cd0efc -> 21f9c6d2c


Remove tests for merge


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a6be102a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a6be102a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a6be102a

Branch: refs/heads/python-sdk
Commit: a6be102a9ebafd3e616ee6aca9a11fbb21c375d9
Parents: 0a66721
Author: Maria Garcia Herrero 
Authored: Tue Nov 22 13:24:08 2016 -0800
Committer: Maria Garcia Herrero 
Committed: Tue Nov 22 13:24:08 2016 -0800

--
 .travis.yml | 9 -
 1 file changed, 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a6be102a/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 7aeb4bc..3080341 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -44,15 +44,6 @@ matrix:
 - os: linux
   env: TEST_PYTHON="1"
 
-# On Linux, run with specific JDKs only.
-- os: linux
-  env: CUSTOM_JDK="oraclejdk8" MAVEN_OVERRIDE="$MAVEN_OVERRIDE 
$MAVEN_CONTAINER_OVERRIDE"
-- os: linux
-  env: CUSTOM_JDK="oraclejdk7" MAVEN_OVERRIDE="$MAVEN_OVERRIDE 
$MAVEN_CONTAINER_OVERRIDE"
-- os: linux
-  env: CUSTOM_JDK="openjdk7" MAVEN_OVERRIDE="$MAVEN_OVERRIDE 
$MAVEN_CONTAINER_OVERRIDE"
-- os: linux
-  env: MAVEN_OVERRIDE="-Peclipse-jdt -DskipTests $MAVEN_OVERRIDE 
$MAVEN_CONTAINER_OVERRIDE" CUSTOM_JDK="oraclejdk8"
 
 before_install:
   - echo 'MAVEN_OPTS="$MAVEN_OPTS -Xmx1024m -XX:MaxPermSize=512m 
-XX:+BytecodeVerificationLocal"' >> ~/.mavenrc

[GitHub] incubator-beam pull request #1423: Remove tests for merge

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1423


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1423

[kenn]

This closes #1423


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/21f9c6d2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/21f9c6d2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/21f9c6d2

Branch: refs/heads/python-sdk
Commit: 21f9c6d2cff052d662326ce73fdcf1fb08504dda
Parents: 3b5cd0e a6be102
Author: Kenneth Knowles 
Authored: Tue Nov 22 15:31:10 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 15:31:10 2016 -0800

--
 .travis.yml | 9 -
 1 file changed, 9 deletions(-)
--

[jira] [Created] (BEAM-1040) Hadoop InputFormat - IO Transform for reads

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-1040:
--

 Summary: Hadoop InputFormat - IO Transform for reads
 Key: BEAM-1040
 URL: https://issues.apache.org/jira/browse/BEAM-1040
 Project: Beam
  Issue Type: Bug
Reporter: Stephen Sisk
Assignee: Stephen Sisk


We should build a IO read transform that will read data from services 
supporting the Hadoop InputFormat  interface [1]

This will make it easy to connect to a variety of databases while still 
providing some aspects of scalability since the InputFormat interface provides 
for parallel reading. 

[1] 
https://hadoop.apache.org/docs/r2.7.3/api/org/apache/hadoop/mapred/InputFormat.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1422: Output Keyed Bundles in GroupAlsoByWindow...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1422


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: This closes #1422

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master b41789e9c -> dd85cad02


This closes #1422


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/dd85cad0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/dd85cad0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/dd85cad0

Branch: refs/heads/master
Commit: dd85cad0284d266fc569ecc6d98653bf8aa5d031
Parents: b41789e 1cec970
Author: Kenneth Knowles 
Authored: Tue Nov 22 15:05:17 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 15:05:17 2016 -0800

--
 .../main/java/org/apache/beam/runners/direct/DirectRunner.java  | 5 -
 .../beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java  | 4 +++-
 2 files changed, 7 insertions(+), 2 deletions(-)
--

[2/2] incubator-beam git commit: Output Keyed Bundles in GroupAlsoByWindowEvaluator

[kenn]

Output Keyed Bundles in GroupAlsoByWindowEvaluator

This allows reuse of keys for downstream serialization.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1cec9702
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1cec9702
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1cec9702

Branch: refs/heads/master
Commit: 1cec9702e62b64252149645627d96889edfeb33e
Parents: b41789e
Author: Thomas Groh 
Authored: Tue Nov 22 14:51:39 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 15:05:17 2016 -0800

--
 .../main/java/org/apache/beam/runners/direct/DirectRunner.java  | 5 -
 .../beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java  | 4 +++-
 2 files changed, 7 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1cec9702/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index 0060e84..cb31947 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -31,6 +31,7 @@ import java.util.Map;
 import java.util.Set;
 import javax.annotation.Nullable;
 import org.apache.beam.runners.core.GBKIntoKeyedWorkItems;
+import org.apache.beam.runners.direct.DirectGroupByKey.DirectGroupAlsoByWindow;
 import org.apache.beam.runners.direct.DirectGroupByKey.DirectGroupByKeyOnly;
 import org.apache.beam.runners.direct.DirectRunner.DirectPipelineResult;
 import 
org.apache.beam.runners.direct.TestStreamEvaluatorFactory.DirectTestStreamFactory;
@@ -310,7 +311,9 @@ public class DirectRunner
 KeyedPValueTrackingVisitor keyedPValueVisitor =
 KeyedPValueTrackingVisitor.create(
 ImmutableSet.>of(
-GroupByKey.class, DirectGroupByKeyOnly.class));
+GBKIntoKeyedWorkItems.class,
+DirectGroupByKeyOnly.class,
+DirectGroupAlsoByWindow.class));
 pipeline.traverseTopologically(keyedPValueVisitor);
 
 DisplayDataValidator.validatePipeline(pipeline);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1cec9702/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
index b946e4d..36c742b 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/GroupAlsoByWindowEvaluatorFactory.java
@@ -112,6 +112,7 @@ class GroupAlsoByWindowEvaluatorFactory implements 
TransformEvaluatorFactory {
 private @SuppressWarnings("unchecked") final WindowingStrategy
 windowingStrategy;
 
+private final StructuralKey structuralKey;
 private final Collection> outputBundles;
 private final ImmutableList.Builder>> 
unprocessedElements;
 private final AggregatorContainer.Mutator aggregatorChanges;
@@ -130,6 +131,7 @@ class GroupAlsoByWindowEvaluatorFactory implements 
TransformEvaluatorFactory {
   this.evaluationContext = evaluationContext;
   this.application = application;
 
+  structuralKey = inputBundle.getKey();
   stepContext = evaluationContext
   .getExecutionContext(application, inputBundle.getKey())
   .getOrCreateStepContext(
@@ -159,7 +161,7 @@ class GroupAlsoByWindowEvaluatorFactory implements 
TransformEvaluatorFactory {
   K key = workItem.key();
 
   UncommittedBundle>> bundle =
-  evaluationContext.createBundle(application.getOutput());
+  evaluationContext.createKeyedBundle(structuralKey, 
application.getOutput());
   outputBundles.add(bundle);
   CopyOnAccessInMemoryStateInternals stateInternals =
   (CopyOnAccessInMemoryStateInternals) stepContext.stateInternals();

[GitHub] incubator-beam pull request #1423: Remove tests for merge

[mariapython]

GitHub user mariapython opened a pull request:

https://github.com/apache/incubator-beam/pull/1423

Remove tests for merge

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mariapython/incubator-beam merge_master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1423


commit a6be102a9ebafd3e616ee6aca9a11fbb21c375d9
Author: Maria Garcia Herrero 
Date:   2016-11-22T21:24:08Z

Remove tests for merge




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1422: Output Keyed Bundles in GroupAlsoByWindow...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1422

Output Keyed Bundles in GroupAlsoByWindowEvaluator

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This allows reuse of keys for downstream serialization.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam no_gbk_keyedness

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1422.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1422


commit 22fbb1728c73d4595fad9bd6e1a13c83100048ca
Author: Thomas Groh 
Date:   2016-11-22T22:51:39Z

Output Keyed Bundles in GroupAlsoByWindowEvaluator

This allows reuse of keys for downstream serialization.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1420: [BEAM-147] Adding Metrics API to Python S...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1420

[BEAM-147] Adding Metrics API to Python SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam metrics-api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1420.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1420


commit d5ced340e83d850d2dfbe1cfed2b28100bf45d80
Author: Pablo 
Date:   2016-11-22T22:31:26Z

Progress on Metrics API




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1025) User guide - "How to create Beam IO Transforms"

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk reassigned BEAM-1025:
--

Assignee: Stephen Sisk

> User guide - "How to create Beam IO Transforms"
> ---
>
> Key: BEAM-1025
> URL: https://issues.apache.org/jira/browse/BEAM-1025
> Project: Beam
>  Issue Type: Task
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> Beam has javadocs for how to create a read or write transform, but no 
> friendly user guide on how to get started using BoundedSource/BoundedReader.
> This should cover:
> * background on beam's source/sink API design 
> * design patterns
> * evaluating different data sources (eg, what are the properties of a pub sub 
> system that affect how you should write your UnboundedSource? What is the 
> best design for reading from a NoSql style source?)
> * testing - how to write unit, integration (and once we have them, 
> performance tests)
> * public API recommendations
> This is related, but not strictly overlapping with: 
> https://issues.apache.org/jira/browse/BEAM-193
> - the Dataflow SDK documentation for "Custom Sources and Sinks"  contains 
> some info about writing Sources/Sinks, but it is somewhat out of date, and 
> doesn't reflect the things we've learned recently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1421: Improve the GCS stat using batchAPI

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/incubator-beam/pull/1421

Improve the GCS stat using batchAPI

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-stat-files-using-batch-api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1421.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1421


commit 6c59b7311946d56cac08dfc156bbeea2c254f021
Author: Sourabh Bajaj 
Date:   2016-11-22T22:36:33Z

Improve the GCS stat using batchAPI




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1027) Hosting data stores to enable IO Transform testing

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk reassigned BEAM-1027:
--

Assignee: Stephen Sisk

> Hosting data stores to enable IO Transform testing
> --
>
> Key: BEAM-1027
> URL: https://issues.apache.org/jira/browse/BEAM-1027
> Project: Beam
>  Issue Type: Task
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> Currently we have a good set of unit tests for our IO Transforms - those
> tend to run against in-memory versions of the data stores. However, we'd
> like to further increase our test coverage to include running them against
> real instances of the data stores that the IO Transforms work against (e.g.
> cassandra, mongodb, kafka, etc…), which means we'll need to have real
> instances of various data stores.
> Additionally, if we want to do performance regression detection, it's
> important to have instances of the services that behave realistically,
> which isn't true of in-memory or dev versions of the services.
> My proposed solution is in 
> https://lists.apache.org/thread.html/367fd9669411f21c9ec1f2d27df60464f49d5ce81e6bd16de401d035@%3Cdev.beam.apache.org%3E
>  
> - it still needs further discussion, and (assuming we agree on the general 
> idea), the beam community needs to decide which cluster management software 
> we want to use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-147) Introduce an easy API for pipeline metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688146#comment-15688146
 ] 

ASF GitHub Bot commented on BEAM-147:
-

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1420

[BEAM-147] Adding Metrics API to Python SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam metrics-api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1420.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1420


commit d5ced340e83d850d2dfbe1cfed2b28100bf45d80
Author: Pablo 
Date:   2016-11-22T22:31:26Z

Progress on Metrics API




> Introduce an easy API for pipeline metrics
> --
>
> Key: BEAM-147
> URL: https://issues.apache.org/jira/browse/BEAM-147
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Robert Bradshaw
>Assignee: Ben Chambers
>
> The existing Aggregators are confusing both because of their name and because 
> they serve multiple purposes.
> Previous discussions around Aggregators/metrics/etc.
> See discussion at 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
>  and 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser
>  . Exact name still being bikeshedded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-147) Introduce an easy API for pipeline metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688141#comment-15688141
 ] 

ASF GitHub Bot commented on BEAM-147:
-

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1419

[BEAM-147] Adding Metrics API to Python SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam metrics-api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1419.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1419


commit 937cf69e958d4a82fb274f311de248930298db69
Author: Robert Bradshaw 
Date:   2016-07-22T21:32:33Z

Fix multi-input named PTransforms.

Now delegate the __ror__ logic entirely for the naming wrapper.

commit 031c4cce0b9eae0d50a49f43ffeced1edbfd2f8f
Author: Robert Bradshaw 
Date:   2016-07-22T21:34:58Z

Move names out of transform constructors.

sed -i -r 's/[|] (\S+)[(](["'"'"'][^"'"'"']+.)(, +|([)]))/| \2 >> \1(\4/g'

Small number of tests will need to be fixed by hand.

commit 7c186ce4ce18c98cf3979b6f0f07c72a47b42ec9
Author: Robert Bradshaw 
Date:   2016-07-22T22:55:26Z

Fix error messages for externally named PTransforms.

commit 362f2e9e4398662d55a2a4e6399f43155c58ed24
Author: Ahmet Altay 
Date:   2016-07-22T23:03:08Z

fix pipeline test

commit 2ff5630be06034e25d60e105bbe1a718ae06b4d6
Author: Chamikara Jayalath 
Date:   2016-07-22T23:04:29Z

Fixes examples

commit 01830510bded3c22ddd96937f5d83547702a4385
Author: Robert Bradshaw 
Date:   2016-07-22T23:05:23Z

Cleanup and fix combiners_test.

commit 2a59a121441a003bad949ae6a23d58a9cf2b3059
Author: Robert Bradshaw 
Date:   2016-07-22T23:24:48Z

Fix label-sensitive test.

commit b15d35ca6e585e75153e05d96403336889cc6894
Author: Robert Bradshaw 
Date:   2016-07-23T01:35:22Z

Lint fixes.

commit c5b5b14d35fc7f6b0a576f0fca19b730015e1282
Author: Robert Bradshaw 
Date:   2016-07-23T08:07:44Z

fixup: failing tests expecting name

commit e3c078fe28553b7e7317316b6df51b4c570573ba
Author: Robert Bradshaw 
Date:   2016-07-23T08:32:23Z

Final cleanup pass.

commit 38d9dea2e62af280e4b9c258cedee70d6bcaa8ca
Author: Robert Bradshaw 
Date:   2016-07-23T23:43:47Z

Closes #718

commit ac3249fe51e9674149ae9cccd48c7290aa8bf34d
Author: Robert Bradshaw 
Date:   2016-07-23T00:33:57Z

Make DoFnRunner a Receiver.

commit d20cf64287fdd22b1efa1a3afe346a0a1d5cf924
Author: Robert Bradshaw 
Date:   2016-07-23T07:37:42Z

Receiver and LoggingContext adapters.

This allows for cdef calls iff the right type is passed in.

commit de35f28294dfb4f47b429635eff843b38279e240
Author: Robert Bradshaw 
Date:   2016-07-23T07:54:11Z

Allow passing logging context directly.

This is better than passing a logger module with a specific class.

commit 153916fe9f641f5fe5e8d473caef67f449ee6bca
Author: Robert Bradshaw 
Date:   2016-07-25T19:02:02Z

Closes #721

commit 8a7bc71dee1e3f9f17c29e5e558870a6bc5f4880
Author: Charles Chen 
Date:   2016-07-26T01:22:08Z

Fix SDK name and version sent to the Cloud Dataflow service

commit 76f3864b61ad9e2ac9a2e81e7a0d5993db5fde6c
Author: Charles Chen 
Date:   2016-07-26T01:36:45Z

Update docstring.

commit 26ff6579531d598e91b220fa21ec9bd7f9220f78
Author: Robert Bradshaw 
Date:   2016-07-26T04:11:31Z

Closes #730

commit 2f4054ba37da1c1100f45a572d96e7a6e2e60152
Author: Charles Chen 
Date:   2016-07-25T18:44:22Z

Make TextFileReader observable

This allows future implementation of size tracking for elements in side 
input sources.

commit 53ab635c75e4b94b3930a601911a717ddc499efe
Author: Robert Bradshaw 
Date:   2016-07-28T18:05:56Z

Closes #726

commit adb3ed93053c83b4e28e7baa879e9aee82f02785
Author: Robert Bradshaw 
Date:   2016-07-27T17:09:49Z

Better top implementation.

When selecting the top k of n, it is common that k << n.
Using a heap is O(n log k) while select algorithms can
achieve O(n + k log k).

This also avoids the ugliness that heapq does not take the
comparator as an 

[jira] [Commented] (BEAM-147) Introduce an easy API for pipeline metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15688142#comment-15688142
 ] 

ASF GitHub Bot commented on BEAM-147:
-

Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1419


> Introduce an easy API for pipeline metrics
> --
>
> Key: BEAM-147
> URL: https://issues.apache.org/jira/browse/BEAM-147
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Robert Bradshaw
>Assignee: Ben Chambers
>
> The existing Aggregators are confusing both because of their name and because 
> they serve multiple purposes.
> Previous discussions around Aggregators/metrics/etc.
> See discussion at 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
>  and 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser
>  . Exact name still being bikeshedded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1419: [BEAM-147] Adding Metrics API to Python S...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1419

[BEAM-147] Adding Metrics API to Python SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam metrics-api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1419.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1419


commit 937cf69e958d4a82fb274f311de248930298db69
Author: Robert Bradshaw 
Date:   2016-07-22T21:32:33Z

Fix multi-input named PTransforms.

Now delegate the __ror__ logic entirely for the naming wrapper.

commit 031c4cce0b9eae0d50a49f43ffeced1edbfd2f8f
Author: Robert Bradshaw 
Date:   2016-07-22T21:34:58Z

Move names out of transform constructors.

sed -i -r 's/[|] (\S+)[(](["'"'"'][^"'"'"']+.)(, +|([)]))/| \2 >> \1(\4/g'

Small number of tests will need to be fixed by hand.

commit 7c186ce4ce18c98cf3979b6f0f07c72a47b42ec9
Author: Robert Bradshaw 
Date:   2016-07-22T22:55:26Z

Fix error messages for externally named PTransforms.

commit 362f2e9e4398662d55a2a4e6399f43155c58ed24
Author: Ahmet Altay 
Date:   2016-07-22T23:03:08Z

fix pipeline test

commit 2ff5630be06034e25d60e105bbe1a718ae06b4d6
Author: Chamikara Jayalath 
Date:   2016-07-22T23:04:29Z

Fixes examples

commit 01830510bded3c22ddd96937f5d83547702a4385
Author: Robert Bradshaw 
Date:   2016-07-22T23:05:23Z

Cleanup and fix combiners_test.

commit 2a59a121441a003bad949ae6a23d58a9cf2b3059
Author: Robert Bradshaw 
Date:   2016-07-22T23:24:48Z

Fix label-sensitive test.

commit b15d35ca6e585e75153e05d96403336889cc6894
Author: Robert Bradshaw 
Date:   2016-07-23T01:35:22Z

Lint fixes.

commit c5b5b14d35fc7f6b0a576f0fca19b730015e1282
Author: Robert Bradshaw 
Date:   2016-07-23T08:07:44Z

fixup: failing tests expecting name

commit e3c078fe28553b7e7317316b6df51b4c570573ba
Author: Robert Bradshaw 
Date:   2016-07-23T08:32:23Z

Final cleanup pass.

commit 38d9dea2e62af280e4b9c258cedee70d6bcaa8ca
Author: Robert Bradshaw 
Date:   2016-07-23T23:43:47Z

Closes #718

commit ac3249fe51e9674149ae9cccd48c7290aa8bf34d
Author: Robert Bradshaw 
Date:   2016-07-23T00:33:57Z

Make DoFnRunner a Receiver.

commit d20cf64287fdd22b1efa1a3afe346a0a1d5cf924
Author: Robert Bradshaw 
Date:   2016-07-23T07:37:42Z

Receiver and LoggingContext adapters.

This allows for cdef calls iff the right type is passed in.

commit de35f28294dfb4f47b429635eff843b38279e240
Author: Robert Bradshaw 
Date:   2016-07-23T07:54:11Z

Allow passing logging context directly.

This is better than passing a logger module with a specific class.

commit 153916fe9f641f5fe5e8d473caef67f449ee6bca
Author: Robert Bradshaw 
Date:   2016-07-25T19:02:02Z

Closes #721

commit 8a7bc71dee1e3f9f17c29e5e558870a6bc5f4880
Author: Charles Chen 
Date:   2016-07-26T01:22:08Z

Fix SDK name and version sent to the Cloud Dataflow service

commit 76f3864b61ad9e2ac9a2e81e7a0d5993db5fde6c
Author: Charles Chen 
Date:   2016-07-26T01:36:45Z

Update docstring.

commit 26ff6579531d598e91b220fa21ec9bd7f9220f78
Author: Robert Bradshaw 
Date:   2016-07-26T04:11:31Z

Closes #730

commit 2f4054ba37da1c1100f45a572d96e7a6e2e60152
Author: Charles Chen 
Date:   2016-07-25T18:44:22Z

Make TextFileReader observable

This allows future implementation of size tracking for elements in side 
input sources.

commit 53ab635c75e4b94b3930a601911a717ddc499efe
Author: Robert Bradshaw 
Date:   2016-07-28T18:05:56Z

Closes #726

commit adb3ed93053c83b4e28e7baa879e9aee82f02785
Author: Robert Bradshaw 
Date:   2016-07-27T17:09:49Z

Better top implementation.

When selecting the top k of n, it is common that k << n.
Using a heap is O(n log k) while select algorithms can
achieve O(n + k log k).

This also avoids the ugliness that heapq does not take the
comparator as an argument, resulting in _HeapItem classes that
were cumbersome and expensive to serialize.

commit 866a09d6c5c4c1c3c39d3a1f2e7b4cb7ec319f6f
Author: Robert Bradshaw 
Date:   2016-07-27T19:23:02Z

Allow Top operations to take key argument rather

[GitHub] incubator-beam pull request #1419: [BEAM-147] Adding Metrics API to Python S...

[pabloem]

Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1419


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: fixing reviewer comments

[kenn]

fixing reviewer comments


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d5a68ea8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d5a68ea8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d5a68ea8

Branch: refs/heads/python-sdk
Commit: d5a68ea839eda941b615e9bfe50107d9c9d7a53c
Parents: 102e677
Author: Ahmet Altay 
Authored: Mon Nov 21 22:27:24 2016 -0800
Committer: Ahmet Altay 
Committed: Mon Nov 21 22:27:24 2016 -0800

--
 .../apache_beam/runners/direct/transform_evaluator.py | 14 +-
 1 file changed, 9 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d5a68ea8/sdks/python/apache_beam/runners/direct/transform_evaluator.py
--
diff --git a/sdks/python/apache_beam/runners/direct/transform_evaluator.py 
b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
index 5a79ab2..7a9a31f 100644
--- a/sdks/python/apache_beam/runners/direct/transform_evaluator.py
+++ b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
@@ -527,9 +527,17 @@ class _NativeWriteEvaluator(_TransformEvaluator):
 self.state.append(element)
 
   def finish_bundle(self):
+# finish_bundle will append incoming bundles in memory until all the 
bundles
+# carrying data is processed. This is done to produce only a single output
+# shard (some tests depends on this behavior). It is possible to have
+# incoming empty bundles after the output is produced, these bundles will 
be
+# ignored and would not generate additional output files.
 # TODO(altay): Do not wait until the last bundle to write in a single 
shard.
 if self._is_final_bundle:
-  if not self._has_already_produced_output:
+  if self._has_already_produced_output:
+# Ignore empty bundles that arrive after the output is produced.
+assert self.state == []
+  else:
 if isinstance(self._sink, io.fileio.NativeTextFileSink):
   assert self._sink.num_shards in (0, 1)
   if self._sink.shard_name_template:
@@ -539,10 +547,6 @@ class _NativeWriteEvaluator(_TransformEvaluator):
 with self._sink.writer() as writer:
   for v in self.state:
 writer.Write(v.value)
-  else:
-# Ignore empty bundles that arrive after the output is produced.
-assert self.state == []
-
   state = None
   hold = WatermarkManager.WATERMARK_POS_INF
 else:

[3/3] incubator-beam git commit: This closes #1384

[kenn]

This closes #1384


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3b5cd0ef
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3b5cd0ef
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3b5cd0ef

Branch: refs/heads/python-sdk
Commit: 3b5cd0efc5f5e4b8fad34ee0d976e5e6ba501065
Parents: b4187bd d5a68ea
Author: Kenneth Knowles 
Authored: Tue Nov 22 13:56:37 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 22 13:56:37 2016 -0800

--
 .../runners/direct/transform_evaluator.py   | 33 ++--
 1 file changed, 23 insertions(+), 10 deletions(-)
--

[GitHub] incubator-beam pull request #1384: Fix the flaky test_model_multiple_pcollec...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1384


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] incubator-beam git commit: Fix the flaky test_model_multiple_pcollections_partition test

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk b4187bd91 -> 3b5cd0efc


Fix the flaky test_model_multiple_pcollections_partition test

_NativeWriteEvaluator should ignore empty bundles that arrive after a
write. Write happens once the last bundle containing data is processed.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/102e6773
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/102e6773
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/102e6773

Branch: refs/heads/python-sdk
Commit: 102e677375869386cc927ba649fddf6736455307
Parents: b83f12b
Author: Ahmet Altay 
Authored: Thu Nov 17 17:48:13 2016 -0800
Committer: Ahmet Altay 
Committed: Thu Nov 17 17:48:13 2016 -0800

--
 .../runners/direct/transform_evaluator.py   | 27 +---
 1 file changed, 18 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/102e6773/sdks/python/apache_beam/runners/direct/transform_evaluator.py
--
diff --git a/sdks/python/apache_beam/runners/direct/transform_evaluator.py 
b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
index 093f183..5a79ab2 100644
--- a/sdks/python/apache_beam/runners/direct/transform_evaluator.py
+++ b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
@@ -513,6 +513,11 @@ class _NativeWriteEvaluator(_TransformEvaluator):
 return (self._execution_context.watermarks.input_watermark
 == WatermarkManager.WATERMARK_POS_INF)
 
+  @property
+  def _has_already_produced_output(self):
+return (self._execution_context.watermarks.output_watermark
+== WatermarkManager.WATERMARK_POS_INF)
+
   def start_bundle(self):
 # state: [values]
 self.state = (self._execution_context.existing_state
@@ -524,15 +529,19 @@ class _NativeWriteEvaluator(_TransformEvaluator):
   def finish_bundle(self):
 # TODO(altay): Do not wait until the last bundle to write in a single 
shard.
 if self._is_final_bundle:
-  if isinstance(self._sink, io.fileio.NativeTextFileSink):
-assert self._sink.num_shards in (0, 1)
-if self._sink.shard_name_template:
-  self._sink.file_path += '-0-of-1'
-  self._sink.file_path += self._sink.file_name_suffix
-  self._sink.pipeline_options = self._evaluation_context.pipeline_options
-  with self._sink.writer() as writer:
-for v in self.state:
-  writer.Write(v.value)
+  if not self._has_already_produced_output:
+if isinstance(self._sink, io.fileio.NativeTextFileSink):
+  assert self._sink.num_shards in (0, 1)
+  if self._sink.shard_name_template:
+self._sink.file_path += '-0-of-1'
+self._sink.file_path += self._sink.file_name_suffix
+self._sink.pipeline_options = self._evaluation_context.pipeline_options
+with self._sink.writer() as writer:
+  for v in self.state:
+writer.Write(v.value)
+  else:
+# Ignore empty bundles that arrive after the output is produced.
+assert self.state == []
 
   state = None
   hold = WatermarkManager.WATERMARK_POS_INF

[jira] [Closed] (BEAM-1039) Spark context is never actually re-used in tests

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin closed BEAM-1039.

   Resolution: Not A Problem
Fix Version/s: Not applicable

> Spark context is never actually re-used in tests
> 
>
> Key: BEAM-1039
> URL: https://issues.apache.org/jira/browse/BEAM-1039
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
> Fix For: Not applicable
>
>
> I think that due to a slight typo 
> {{Boolean.getBoolean(TEST_REUSE_SPARK_CONTEXT)}} got mixed up with
> {{Boolean.getBoolean(System.getProperty(TEST_REUSE_SPARK_CONTEXT))}} in the 
> following 2 places, which may have caused {{SparkContext}} to behave not 
> quite as expected.
> *  
> https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L52
> *  
> https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L70
> I'll take care of it as part of something I'm working on, so a PR should 
> follow shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1039) Spark context is never actually re-used in tests

[Stas Levin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15687906#comment-15687906
 ] 

Stas Levin commented on BEAM-1039:
--

For some reason it looked like...

Must have been a brain f... My bad, closing.

> Spark context is never actually re-used in tests
> 
>
> Key: BEAM-1039
> URL: https://issues.apache.org/jira/browse/BEAM-1039
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Stas Levin
>Assignee: Stas Levin
>
> I think that due to a slight typo 
> {{Boolean.getBoolean(TEST_REUSE_SPARK_CONTEXT)}} got mixed up with
> {{Boolean.getBoolean(System.getProperty(TEST_REUSE_SPARK_CONTEXT))}} in the 
> following 2 places, which may have caused {{SparkContext}} to behave not 
> quite as expected.
> *  
> https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L52
> *  
> https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java#L70
> I'll take care of it as part of something I'm working on, so a PR should 
> follow shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1020:
---
Fix Version/s: (was: Not applicable)
   0.4.0-incubating

> FileBasedSink fails to remove temporary directory on Windows operating system
> -
>
> Key: BEAM-1020
> URL: https://issues.apache.org/jira/browse/BEAM-1020
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Tenghuan He
>Assignee: Tenghuan He
> Fix For: 0.4.0-incubating
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
>  Beam fails to remove temporary directory after completing pipelines and 
> sinking output results on Windows operating system and throws the following 
> exception
> {noformat} 
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
>   at 
> org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
>   at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
>   at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
>   at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
>   at 
> org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
>   at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
> {noformat}
> This is due to the use of wildcard * when matching all files in the temporary 
> directory, while Windows does not support wildcard * in the file path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1020) FileBasedSink fails to remove temporary directory on Windows operating system

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1020:
---
Assignee: Tenghuan He  (was: Davor Bonaci)

> FileBasedSink fails to remove temporary directory on Windows operating system
> -
>
> Key: BEAM-1020
> URL: https://issues.apache.org/jira/browse/BEAM-1020
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Tenghuan He
>Assignee: Tenghuan He
> Fix For: Not applicable
>
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
>  Beam fails to remove temporary directory after completing pipelines and 
> sinking output results on Windows operating system and throws the following 
> exception
> {noformat} 
> Exception in thread "main" 
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
> java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:348)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:73)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:179)
>   at 
> org.apache.beam.examples.MinimalWordCount.main(MinimalWordCount.java:116)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: java.nio.file.InvalidPathException: Illegal char <*> at index 0: *
>   at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
>   at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
>   at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
>   at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
>   at sun.nio.fs.AbstractPath.resolve(AbstractPath.java:53)
>   at 
> org.apache.beam.sdk.util.FileIOChannelFactory.resolve(FileIOChannelFactory.java:159)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:469)
>   at 
> org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:388)
>   at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
> {noformat}
> This is due to the use of wildcard * when matching all files in the temporary 
> directory, while Windows does not support wildcard * in the file path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)