Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #516

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323753#comment-15323753
 ] 

ASF GitHub Bot commented on BEAM-22:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/392


> DirectPipelineRunner: support for unbounded collections
> ---
>
> Key: BEAM-22
> URL: https://issues.apache.org/jira/browse/BEAM-22
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Davor Bonaci
>Assignee: Thomas Groh
>
> DirectPipelineRunner currently runs over bounded PCollections only, and 
> implements only a portion of the Beam Model.
> We should improve it to faithfully implement the full Beam Model, such as add 
> ability to run over unbounded PCollections, and better resemble execution 
> model in a distributed system.
> This further enables features such as a testing source which may simulate 
> late data and test triggers in the pipeline. Finally, we may want to expose 
> an option to select between "debug" (single threaded), "chaos monkey" (test 
> as many model requirements as possible), and "performance" (multi-threaded).
> more testing (chaos monkey) 
> Once this is done, we should update this StackOverflow question:
> http://stackoverflow.com/questions/35350113/testing-triggers-with-processing-time/35401426#35401426



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] incubator-beam git commit: Execute NeedsRunner tests in the Direct Runner

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master d53e96a0d -> 829121960


Execute NeedsRunner tests in the Direct Runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9e797e3d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9e797e3d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9e797e3d

Branch: refs/heads/master
Commit: 9e797e3d3a6bda2ff42b8c3acf8fd60fec179aa4
Parents: ee1297e
Author: Thomas Groh 
Authored: Fri May 20 15:18:11 2016 -0700
Committer: Thomas Groh 
Committed: Thu Jun 9 15:36:48 2016 -0700

--
 runners/direct-java/pom.xml | 25 -
 1 file changed, 24 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9e797e3d/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index fb2d2e1..def7207 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -86,7 +86,7 @@
   test
 
 
-  org.apache.beam.sdk.testing.RunnableOnService
+  org.apache.beam.sdk.testing.NeedsRunner
   none
   true
   
@@ -340,5 +340,28 @@
   test-jar
   test
 
+
+
+
+  org.codehaus.woodstox
+  stax2-api
+  ${stax2.version}
+  test
+
+
+
+  org.codehaus.woodstox
+  woodstox-core-asl
+  ${woodstox.version}
+  
+
+
+  javax.xml.stream
+  stax-api
+
+  
+  test
+
+
   
 

[3/3] incubator-beam git commit: This closes #392

[kenn]

This closes #392


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/82912196
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/82912196
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/82912196

Branch: refs/heads/master
Commit: 82912196080b7ae79b39e355d10077ad8654eda6
Parents: d53e96a 9e797e3
Author: Kenn Knowles 
Authored: Thu Jun 9 19:26:23 2016 -0700
Committer: Kenn Knowles 
Committed: Thu Jun 9 19:26:23 2016 -0700

--
 runners/direct-java/pom.xml | 25 +++-
 .../direct/InProcessEvaluationContext.java  |  5 ++--
 .../direct/WatermarkCallbackExecutor.java   | 10 
 3 files changed, 32 insertions(+), 8 deletions(-)
--

[2/3] incubator-beam git commit: Use a DirectExecutor for Watermark Callbacks

[kenn]

Use a DirectExecutor for Watermark Callbacks

This fixes a resource leak where the executor service is not properly
shut down with the rest of the DirectRunner.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ee1297e2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ee1297e2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ee1297e2

Branch: refs/heads/master
Commit: ee1297e21a481fbea52475c0732526a0441d03cb
Parents: d53e96a
Author: Thomas Groh 
Authored: Tue Jun 7 17:50:38 2016 -0700
Committer: Thomas Groh 
Committed: Thu Jun 9 15:36:48 2016 -0700

--
 .../beam/runners/direct/InProcessEvaluationContext.java   |  5 +++--
 .../beam/runners/direct/WatermarkCallbackExecutor.java| 10 +-
 2 files changed, 8 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ee1297e2/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
index 981a842..db8baa0 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessEvaluationContext.java
@@ -46,13 +46,13 @@ import org.apache.beam.sdk.values.PValue;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Iterables;
+import com.google.common.util.concurrent.MoreExecutors;
 
 import java.util.Collection;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.Executors;
 
 import javax.annotation.Nullable;
 
@@ -130,7 +130,8 @@ class InProcessEvaluationContext {
 this.applicationStateInternals = new ConcurrentHashMap<>();
 this.mergedCounters = new CounterSet();
 
-this.callbackExecutor = 
WatermarkCallbackExecutor.create(Executors.newSingleThreadExecutor());
+this.callbackExecutor =
+WatermarkCallbackExecutor.create(MoreExecutors.directExecutor());
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ee1297e2/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
index 1c9b050..0f73b1d 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkCallbackExecutor.java
@@ -29,7 +29,7 @@ import org.joda.time.Instant;
 import java.util.PriorityQueue;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executor;
 
 /**
  * Executes callbacks that occur based on the progression of the watermark 
per-step.
@@ -51,15 +51,15 @@ class WatermarkCallbackExecutor {
   /**
* Create a new {@link WatermarkCallbackExecutor}.
*/
-  public static WatermarkCallbackExecutor create(ExecutorService executor) {
+  public static WatermarkCallbackExecutor create(Executor executor) {
 return new WatermarkCallbackExecutor(executor);
   }
 
   private final ConcurrentMap
   callbacks;
-  private final ExecutorService executor;
+  private final Executor executor;
 
-  private WatermarkCallbackExecutor(ExecutorService executor) {
+  private WatermarkCallbackExecutor(Executor executor) {
 this.callbacks = new ConcurrentHashMap<>();
 this.executor = executor;
   }
@@ -101,7 +101,7 @@ class WatermarkCallbackExecutor {
 }
 synchronized (callbackQueue) {
   while (!callbackQueue.isEmpty() && 
callbackQueue.peek().shouldFire(watermark)) {
-executor.submit(callbackQueue.poll().getCallback());
+executor.execute(callbackQueue.poll().getCallback());
   }
 }
   }

[GitHub] incubator-beam pull request #392: [BEAM-22][BEAM-243] Execute NeedsRunner te...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/392


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #515

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #514

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #443: Expand set of dependencies to scan for Run...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/443


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk [created] d53e96a0d

[GitHub] incubator-beam pull request #406: [BEAM-155] Use custom Assertions in Spark ...

[tgroh]

Github user tgroh closed the pull request at:

https://github.com/apache/incubator-beam/pull/406


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/4] incubator-beam git commit: Fix TriggerExampleTest

[kenn]

Fix TriggerExampleTest


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/77aa0938
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/77aa0938
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/77aa0938

Branch: refs/heads/master
Commit: 77aa0938f75f9f3d18a4fa79a5ffe6159167f4d5
Parents: 810ffeb
Author: Kenneth Knowles 
Authored: Wed Jun 8 16:22:34 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 9 14:41:09 2016 -0700

--
 .../examples/cookbook/TriggerExampleTest.java   | 61 +---
 1 file changed, 41 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/77aa0938/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java
 
b/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java
index fe75d14..cddce7f 100644
--- 
a/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java
+++ 
b/examples/java/src/test/java/org/apache/beam/examples/cookbook/TriggerExampleTest.java
@@ -34,6 +34,8 @@ import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TimestampedValue;
 
 import com.google.api.services.bigquery.model.TableRow;
+import com.google.common.base.Joiner;
+import com.google.common.collect.Lists;
 
 import org.joda.time.Duration;
 import org.joda.time.Instant;
@@ -44,7 +46,9 @@ import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
 
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
+import java.util.Map;
 
 /**
  * Unit Tests for {@link TriggerExample}.
@@ -70,21 +74,27 @@ public class TriggerExampleTest {
   + 
"0.001,74.8,1,9,3,0.0028,71,1,9,12,0.0099,97.4,1,9,13,0.0121,50.0,1,0,0"
   + ",0,0", new Instant(1)));
 
-  private static final TableRow OUT_ROW_1 = new TableRow()
-  .set("trigger_type", "default")
-  .set("freeway", "5").set("total_flow", 30)
-  .set("number_of_records", 1)
-  .set("isFirst", true).set("isLast", true)
-  .set("timing", "ON_TIME")
-  .set("window", "[1970-01-01T00:01:00.000Z..1970-01-01T00:02:00.000Z)");
-
-  private static final TableRow OUT_ROW_2 = new TableRow()
-  .set("trigger_type", "default")
-  .set("freeway", "110").set("total_flow", 90)
-  .set("number_of_records", 2)
-  .set("isFirst", true).set("isLast", true)
-  .set("timing", "ON_TIME")
-  .set("window", "[1970-01-01T00:00:00.000Z..1970-01-01T00:01:00.000Z)");
+  private static final TableRow OUT_ROW_1 =
+  new TableRow()
+  .set("trigger_type", "default")
+  .set("freeway", "5")
+  .set("total_flow", 30)
+  .set("number_of_records", 1)
+  .set("isFirst", true)
+  .set("isLast", true)
+  .set("timing", "ON_TIME")
+  .set("window", 
"[1970-01-01T00:01:00.000Z..1970-01-01T00:02:00.000Z)");
+
+  private static final TableRow OUT_ROW_2 =
+  new TableRow()
+  .set("trigger_type", "default")
+  .set("freeway", "110")
+  .set("total_flow", 90)
+  .set("number_of_records", 2)
+  .set("isFirst", true)
+  .set("isLast", true)
+  .set("timing", "ON_TIME")
+  .set("window", 
"[1970-01-01T00:00:00.000Z..1970-01-01T00:01:00.000Z)");
 
   @Test
   public void testExtractTotalFlow() throws Exception {
@@ -112,15 +122,26 @@ public class TriggerExampleTest {
 .apply(Window.>into(FixedWindows.of(Duration.standardMinutes(1
 .apply(new TotalFlow("default"));
 
-PCollection results =  totalFlow.apply(ParDo.of(new 
FormatResults()));
+PCollection results =  totalFlow.apply(ParDo.of(new 
FormatResults()));
 
-
-PAssert.that(results).containsInAnyOrder(OUT_ROW_1, OUT_ROW_2);
+PAssert.that(results)
+.containsInAnyOrder(canonicalFormat(OUT_ROW_1), 
canonicalFormat(OUT_ROW_2));
 pipeline.run();
 
   }
 
-  static class FormatResults extends DoFn {
+  // Sort the fields and toString() the values, since TableRow has a bit of a 
dynamically
+  // typed API and equals()/hashCode() are not appropriate for matching in 
tests
+  static String canonicalFormat(TableRow row) {
+List entries = Lists.newArrayListWithCapacity(row.size());
+for (Map.Entry entry : row.entrySet()) {
+  entries.add(entry.getKey() + ":" + entry.getValue());
+}
+Collections.sort(entries);
+return Joiner.on(",").join(entries);
+  }
+
+  static class FormatResults extends 

[3/4] incubator-beam git commit: Switch Spark streaming tests to custom assertions

[kenn]

Switch Spark streaming tests to custom assertions

The current use of PAssert in the streaming tests for
the Spark runner work via coincidence. PAssert does not
truly support non-global windowing. The switch from
side inputs to GBK, with no change in semantics but hopefully
an easier on-ramp for new runners, incidentally broke
these tests. Soon, PAssert will support windowing, triggers,
and unbounded PCollections. Until then, this change
writes a slightly custom assertion transform for these tests.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f222df10
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f222df10
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f222df10

Branch: refs/heads/master
Commit: f222df109e773f23e56f9e830454356893989a15
Parents: 77aa093
Author: Kenneth Knowles 
Authored: Wed Jun 8 18:11:07 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 9 14:41:09 2016 -0700

--
 .../streaming/FlattenStreamingTest.java |  7 +---
 .../streaming/KafkaStreamingTest.java   | 13 ++-
 .../streaming/SimpleStreamingWordCountTest.java | 18 +++---
 .../streaming/utils/PAssertStreaming.java   | 36 +++-
 4 files changed, 43 insertions(+), 31 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f222df10/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/FlattenStreamingTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/FlattenStreamingTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/FlattenStreamingTest.java
index 15b2f39..976c7c2 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/FlattenStreamingTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/FlattenStreamingTest.java
@@ -25,9 +25,7 @@ import 
org.apache.beam.runners.spark.translation.streaming.utils.PAssertStreamin
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.transforms.Flatten;
-import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.values.PCollection;
@@ -77,13 +75,10 @@ public class FlattenStreamingTest {
 PCollectionList list = 
PCollectionList.of(windowedW1).and(windowedW2);
 PCollection union = list.apply(Flatten.pCollections());
 
-PAssert.thatIterable(union.apply(View.asIterable()))
-.containsInAnyOrder(EXPECTED_UNION);
+PAssertStreaming.assertContents(union, EXPECTED_UNION);
 
 EvaluationResult res = SparkPipelineRunner.create(options).run(p);
 res.close();
-
-PAssertStreaming.assertNoFailures(res);
   }
 
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f222df10/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
index fd75e74..53293fb 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/KafkaStreamingTest.java
@@ -27,17 +27,14 @@ import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 
 import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.ImmutableSet;
 
 import org.apache.kafka.clients.producer.KafkaProducer;
 import org.apache.kafka.clients.producer.ProducerRecord;
@@ -52,7 +49,6 @@ import java.io.IOException;
 import java.util.Collections;
 import java.util.Map;
 import java.util.Properties;
-import 

[4/4] incubator-beam git commit: Base PAssert on GBK instead of side inputs

[kenn]

Base PAssert on GBK instead of side inputs

Previously PAssert - hence all RunnableOnService/NeedsRunner
tests - required side input support. This created a very steep
on ramp for new runners.

GroupByKey is a bit more fundamental and most backends will be
able to group by key in the global window very quickly. So switching
the primitive used to gather all the contents of a PCollection for
assertions should make it a bit easier to get early feedback during
runner development.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/810ffeb2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/810ffeb2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/810ffeb2

Branch: refs/heads/master
Commit: 810ffeb2785bf996001c8fadb992410d1f9409c6
Parents: d6adbbf
Author: Kenneth Knowles 
Authored: Wed Jun 8 15:07:52 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 9 14:41:09 2016 -0700

--
 .../testing/TestDataflowPipelineRunner.java |   3 +-
 .../org/apache/beam/sdk/testing/PAssert.java| 737 +--
 .../apache/beam/sdk/testing/PAssertTest.java|  27 -
 3 files changed, 362 insertions(+), 405 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/810ffeb2/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
index 3e8d903..c940e9a 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
@@ -166,7 +166,8 @@ public class TestDataflowPipelineRunner extends 
PipelineRunner OutputT apply(
   PTransform transform, InputT input) {
 if (transform instanceof PAssert.OneSideInputAssert
-|| transform instanceof PAssert.TwoSideInputAssert) {
+|| transform instanceof PAssert.GroupThenAssert
+|| transform instanceof PAssert.GroupThenAssertForSingleton) {
   expectedNumberOfAssertions += 1;
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/810ffeb2/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
index c2cd598..b10c1cb 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
@@ -34,11 +34,14 @@ import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.transforms.Sum;
+import org.apache.beam.sdk.transforms.Values;
 import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.WithKeys;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.util.CoderUtils;
@@ -48,32 +51,27 @@ import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.PDone;
 
-import com.google.common.base.Optional;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Lists;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.io.IOException;
 import java.io.Serializable;
 import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
-import java.util.List;
 import java.util.Map;
 import java.util.NoSuchElementException;
 
 /**
- * An assertion on the contents of a {@link PCollection}
- * incorporated into the pipeline.  Such an assertion
- * can be checked no matter what kind of {@link PipelineRunner} is
- * used.
+ * An assertion on the contents of a {@link PCollection} incorporated into the 
pipeline. Such an
+ * assertion can be checked no matter what kind of {@link PipelineRunner} is 
used.
  *
- * 

[1/4] incubator-beam git commit: Spark runner: Assign windows when re-windowing into global window

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master d6adbbf96 -> d53e96a0d


Spark runner: Assign windows when re-windowing into global window

Previously, window assignment was elided when the window was the
global window. But when the source windows are not the global window,
this elision is not correct. Now window assignment is run except
when both the source *and* the destination window are the global window
(which remains a common case in globally windowed batch tests
using PAssert).


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d53e96a0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d53e96a0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d53e96a0

Branch: refs/heads/master
Commit: d53e96a0d1f5f26ad0e3efc90dc9f7b53135443b
Parents: f222df1
Author: Kenneth Knowles 
Authored: Thu Jun 9 09:15:39 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 9 14:41:09 2016 -0700

--
 .../beam/runners/spark/translation/TransformTranslator.java   | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d53e96a0/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
index b462d35..ebceb6b 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
@@ -699,13 +699,16 @@ public final class TransformTranslator {
 JavaRDDLike inRDD =
 (JavaRDDLike) context.getInputRDD(transform);
 WindowFn windowFn = WINDOW_FG.get("windowFn", transform);
-if (windowFn instanceof GlobalWindows) {
+// Avoid running assign windows if both source and destination are 
global window
+if (context.getInput(transform).getWindowingStrategy().getWindowFn()
+instanceof GlobalWindows
+&& windowFn instanceof GlobalWindows) {
   context.setOutputRDD(transform, inRDD);
 } else {
   @SuppressWarnings("unchecked")
   DoFn addWindowsDoFn = new AssignWindowsDoFn<>(windowFn);
   DoFnFunction dofn =
-  new DoFnFunction<>(addWindowsDoFn, 
context.getRuntimeContext(), null);
+  new DoFnFunction<>(addWindowsDoFn, context.getRuntimeContext(), 
null);
   context.setOutputRDD(transform, inRDD.mapPartitions(dofn));
 }
   }

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #513

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #443: Expand set of dependencies to scan for Run...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/443

Expand set of dependencies to scan for RunnableOnService tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

I'm afraid we may have gotten off easy on `RunnableOnService` since the SDK 
was split into bits and the tests were only pulled in from the core.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam runnable-on-service

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #443


commit 0b953a5b0dd96434edbf4873bf35b4de7e0c2d40
Author: Kenneth Knowles 
Date:   2016-06-09T21:07:54Z

Expand set of dependencies to scan for RunnableOnService tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-332) Matcher(s) for TableRow

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-332:


 Summary: Matcher(s) for TableRow
 Key: BEAM-332
 URL: https://issues.apache.org/jira/browse/BEAM-332
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-core
Reporter: Kenneth Knowles
Priority: Minor


{{TableRow}} has poorly behaved equality, and a certain amount of automatic 
coercion, so matchers based equals() are not applicable in tests. It would be 
handy to have matchers such as "isTableRowEqualTo(otherRow)".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #442: Add BigDecimalCoder and BigIntegerCoder

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/442

Add BigDecimalCoder and BigIntegerCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam bignums

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/442.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #442


commit 252132383e906c107e11f98f53da743b50e66cf2
Author: Jesse Anderson 
Date:   2016-05-09T17:05:15Z

Added BigDecimal coder and tests.

commit 4cd11e079772a42767cd6014016e5a2b4a9b504c
Author: Kenneth Knowles 
Date:   2016-06-09T20:24:28Z

Touch up BigDecimalCoder and tests

commit 41719a7c610eaf6ee4560479ea93e183b7a65ca4
Author: Kenneth Knowles 
Date:   2016-06-09T20:24:49Z

Add BigIntegerCoder and tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #512

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323189#comment-15323189
 ] 

ASF GitHub Bot commented on BEAM-22:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/435


> DirectPipelineRunner: support for unbounded collections
> ---
>
> Key: BEAM-22
> URL: https://issues.apache.org/jira/browse/BEAM-22
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Davor Bonaci
>Assignee: Thomas Groh
>
> DirectPipelineRunner currently runs over bounded PCollections only, and 
> implements only a portion of the Beam Model.
> We should improve it to faithfully implement the full Beam Model, such as add 
> ability to run over unbounded PCollections, and better resemble execution 
> model in a distributed system.
> This further enables features such as a testing source which may simulate 
> late data and test triggers in the pipeline. Finally, we may want to expose 
> an option to select between "debug" (single threaded), "chaos monkey" (test 
> as many model requirements as possible), and "performance" (multi-threaded).
> more testing (chaos monkey) 
> Once this is done, we should update this StackOverflow question:
> http://stackoverflow.com/questions/35350113/testing-triggers-with-processing-time/35401426#35401426



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #435

[kenn]

This closes #435


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d6adbbf9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d6adbbf9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d6adbbf9

Branch: refs/heads/master
Commit: d6adbbf965c974585a4a1165e718061fc54d1cb1
Parents: 983d467 5911347
Author: Kenneth Knowles 
Authored: Thu Jun 9 12:43:18 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jun 9 12:43:18 2016 -0700

--
 .../direct/ExecutorServiceParallelExecutor.java| 17 +++--
 1 file changed, 11 insertions(+), 6 deletions(-)
--

[GitHub] incubator-beam pull request #435: [BEAM-22] Shutdown the InProcessPipelineRu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/435


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Shutdown the InProcessPipelineRunner after Terminating Abnormally

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 983d467a7 -> d6adbbf96


Shutdown the InProcessPipelineRunner after Terminating Abnormally

Ensure that the executor service is shutdown, and the monitor is not
rescheduled, after an exception is thrown.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/59113474
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/59113474
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/59113474

Branch: refs/heads/master
Commit: 591134749f2d24a7d6550e0bb00845a36cdb1616
Parents: 2173000
Author: Thomas Groh 
Authored: Wed Jun 8 12:00:05 2016 -0700
Committer: Thomas Groh 
Committed: Wed Jun 8 12:00:05 2016 -0700

--
 .../direct/ExecutorServiceParallelExecutor.java| 17 +++--
 1 file changed, 11 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/59113474/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
index 3129145..980d764 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
@@ -351,6 +351,8 @@ final class ExecutorServiceParallelExecutor implements 
InProcessExecutor {
 evaluationContext.getPipelineOptions().getAppName(),
 ExecutorServiceParallelExecutor.class.getSimpleName());
 
+private boolean exceptionThrown = false;
+
 @Override
 public void run() {
   String oldName = Thread.currentThread().getName();
@@ -366,6 +368,7 @@ final class ExecutorServiceParallelExecutor implements 
InProcessExecutor {
 scheduleConsumers(update);
   } else if (update.getException().isPresent()) {
 
visibleUpdates.offer(VisibleExecutorUpdate.fromThrowable(update.getException().get()));
+exceptionThrown = true;
   }
   if (System.nanoTime() - updatesStart > 
maxTimeProcessingUpdatesNanos) {
 break;
@@ -434,15 +437,17 @@ final class ExecutorServiceParallelExecutor implements 
InProcessExecutor {
 }
 
 private boolean shouldShutdown() {
-  if (evaluationContext.isDone()) {
-LOG.debug("Pipeline is finished. Shutting down. {}");
-while (!visibleUpdates.offer(VisibleExecutorUpdate.finished())) {
-  visibleUpdates.poll();
+  boolean shouldShutdown = exceptionThrown || evaluationContext.isDone();
+  if (shouldShutdown) {
+if (evaluationContext.isDone()) {
+  LOG.debug("Pipeline is finished. Shutting down. {}");
+  while (!visibleUpdates.offer(VisibleExecutorUpdate.finished())) {
+visibleUpdates.poll();
+  }
 }
 executorService.shutdown();
-return true;
   }
-  return false;
+  return shouldShutdown;
 }
 
 /**

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #511

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-331) WindowedWordCount.AddTimestampFn has nondeterministic timestamp bounds

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323056#comment-15323056
 ] 

ASF GitHub Bot commented on BEAM-331:
-

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/441

[BEAM-331] Make AddTimestampFn range deterministic

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The timestamps added in the WindowedWordCount example are currently
based on when the bundles are executed, which makes the min/max bounds
non-deterministic. This change makes the range  based on the
construction time.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam wwc-addtimestampfn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/441.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #441


commit 6b1d3a7d5a4e2bb9ecb010102a7d0506cda15986
Author: Scott Wegner 
Date:   2016-06-09T18:31:23Z

Make AddTimestampFn range deterministic

The timestamps added in the WindowedWordCount example are currently
based on when the bundles are executed, which makes the min/max bounds
non-deterministic. This change makes the range  based on the
construction time.




> WindowedWordCount.AddTimestampFn has nondeterministic timestamp bounds
> --
>
> Key: BEAM-331
> URL: https://issues.apache.org/jira/browse/BEAM-331
> Project: Beam
>  Issue Type: Bug
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> The timestamps added in the WindowedWordCount example are based on when the 
> bundles are executed, which makes the min/max bounds non-deterministic. 
> It would be more desirable to capture the min/max at construction time. We 
> would like this to be able to use this as an example of adding display data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #441: [BEAM-331] Make AddTimestampFn range deter...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/441

[BEAM-331] Make AddTimestampFn range deterministic

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The timestamps added in the WindowedWordCount example are currently
based on when the bundles are executed, which makes the min/max bounds
non-deterministic. This change makes the range  based on the
construction time.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam wwc-addtimestampfn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/441.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #441


commit 6b1d3a7d5a4e2bb9ecb010102a7d0506cda15986
Author: Scott Wegner 
Date:   2016-06-09T18:31:23Z

Make AddTimestampFn range deterministic

The timestamps added in the WindowedWordCount example are currently
based on when the bundles are executed, which makes the min/max bounds
non-deterministic. This change makes the range  based on the
construction time.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-331) WindowedWordCount.AddTimestampFn has nondeterministic timestamp bounds

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-331:
-

 Summary: WindowedWordCount.AddTimestampFn has nondeterministic 
timestamp bounds
 Key: BEAM-331
 URL: https://issues.apache.org/jira/browse/BEAM-331
 Project: Beam
  Issue Type: Bug
Reporter: Scott Wegner
Assignee: Scott Wegner
Priority: Minor


The timestamps added in the WindowedWordCount example are based on when the 
bundles are executed, which makes the min/max bounds non-deterministic. 

It would be more desirable to capture the min/max at construction time. We 
would like this to be able to use this as an example of adding display data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #510

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #440: [BEAM-???] Modify range tracker to use fir...

[ianzhou1]

GitHub user ianzhou1 opened a pull request:

https://github.com/apache/incubator-beam/pull/440

[BEAM-???] Modify range tracker to use first response as start key

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ianzhou1/incubator-beam beam

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/440.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #440


commit 5e1fc31c5b666ca50105a5588f7f7f68c6f527b7
Author: Aljoscha Krettek 
Date:   2016-05-18T15:46:34Z

[BEAM-295] Remove erroneous close() calls in Flink Create Sources

Collector.close() should only be called by internal Flink components,
not by user functions.

commit c20c9ef6b978bda01c7358c9674ba3e4f11dd3c5
Author: Mark Shields 
Date:   2016-05-31T21:44:20Z

Forward port changes to GC holds

Forward port changes to BEAM/pull/391 from DataflowJavaSDK/pull/289.

commit af794e1afe8f2b22432e7e0285a45f157bdac969
Author: Jean-Baptiste Onofré 
Date:   2016-05-31T19:46:46Z

[BEAM-242] Set javadoc less strict around missing return or param statements

commit df0633d7b5105206928d7d30ccf62410388e46a3
Author: Davor Bonaci 
Date:   2016-06-01T20:13:02Z

Update pom.xml files formatting

This makes it easier to run maven-release-plugin and understand the output. 
Otherwise, the release plugin will make such changes repeatedly in a release 
branch.

commit 06690614e58dfae6f9509dd01cf69df59bd94f36
Author: Davor Bonaci 
Date:   2016-06-01T21:14:08Z

[maven-release-plugin] prepare branch release-0.1.0-incubating

commit f8cbd4d78c9b6aced6ad1375684b60707ce1c6e9
Author: Thomas Groh 
Date:   2016-06-01T21:28:18Z

Use Structural Value keys instead of User Values

This fixes problems with lookup by basing entirely on structural
equality.

commit 1f3dfa288f526335f52cbce9c36f9e3dd28541ff
Author: Davor Bonaci 
Date:   2016-06-01T22:21:41Z

[maven-release-plugin] prepare for next development iteration

commit 5c6e980098486f404df87ff080184c0ede521671
Author: JianfengQian 
Date:   2016-05-21T10:32:29Z

[BEAM-297] Update Flink README.md to  0.2.0-incubating-SNAPSHOT

commit 801f2d7768aba2efc12b14ff6f0d9716ed2ed1b0
Author: Thomas Groh 
Date:   2016-05-30T18:04:15Z

Return from awaitCompletion if Already Done

This ensures that a call to ExecutorService#awaitCompletion returns 
immediately
if there are no visible updates and the executor has completed. If the 
executor is
in this state, no additional visible updates will be published and the call 
will hang.

This sequence generally will not happen, as calls via 
InProcessPipelineResult
return if the state is already terminal, but this ensures that parallel 
calls to
awaitCompletion do not hang one calling thread.

commit 8524c745eefec73c734c336cd44db18649d53f9b
Author: Davor Bonaci 
Date:   2016-06-02T20:21:31Z

Remove reporting section from Spark's pom.xml file

This is currently triggering Maven warnings, due to a missing version 
qualifier. However, we don't run 'site', so this isn't being used at this 
point. Hence, removing instead of adding the version qualifier.

commit 74ba140139b4df97182e4b4bece2a75a09fc5824
Author: Davor Bonaci 
Date:   2016-06-02T22:09:53Z

Move 'travis/' directory into 'testing/travis/'

commit a9fbba9fabf1897b35ad23573f4e276b22f09a6e
Author: Jean-Baptiste Onofré 
Date:   2016-06-03T11:05:58Z

[BEAM-319][BEAM-320] Add distributions module responsible of packaging 
source and binaries distributions.

commit d2172de8d8f320837fb80a7ea101bf16c8accbe5
Author: Jean-Baptiste Onofré 
Date:   2016-06-03T11:55:37Z

Use zip source distribution only (tar.gz doesn't work on Jenkins due to 
UTF-8 encoding and long path name)

commit 459ecf7ae2e3c643e9ee96111b55acfe4c848f13
Author: Raghu Angadi 
Date:   2016-06-04T00:46:38Z

squash PR 

[jira] [Commented] (BEAM-330) Maven exec warning message on Dataflow WordCount Example

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322954#comment-15322954
 ] 

ASF GitHub Bot commented on BEAM-330:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/439


> Maven exec warning message on Dataflow WordCount Example
> 
>
> Key: BEAM-330
> URL: https://issues.apache.org/jira/browse/BEAM-330
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> When running the WordCount example with DataflowPipelineRunner, sometimes the 
> Maven process will emit a warning when trying to tear down the process:
> {quote}{noformat}
> [DEBUG] interrupting thread 
> Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount]
> [DEBUG] joining on thread 
> Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount]
> [WARNING] thread Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount] 
> was interrupted but is still alive after waiting at least 14999msecs
> [WARNING] thread Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount] 
> will linger despite being asked to die via interruption
> [WARNING] NOTE: 1 thread(s) did not finish despite being asked to  via 
> interruption. This is not a problem with exec:java, it is a problem with the 
> running code. Although not serious, it should be remedied.
> [WARNING] Couldn't destroy threadgroup 
> org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=org.apache.beam.examples.WordCount,maxpri=10]
> {noformat}{quote}
> This appears to be some bad interaction between exec-maven-plugin and 
> DataflowPipelineRunner, possibly due to exec-maven-plugin's use of Guice and 
> our shading of it.
> The problem doesn't always reproduce, except when executing multiple Maven 
> targets, such as "mvn install exec:java \[...\]"
> Disabling exec:java's cleanupDaemonThreads indeed suppresses the issue.  I 
> recommend we add this configuration to the root pom.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #439: [BEAM-330] Disable exec-maven-plugin clean...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/439


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: [BEAM-330] Disable exec-maven-plugin cleanupDaemonThreads

[lcwik]

[BEAM-330] Disable exec-maven-plugin cleanupDaemonThreads

This closes #439


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/983d467a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/983d467a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/983d467a

Branch: refs/heads/master
Commit: 983d467a798458e7ee4f73cd8263e6044903bd0e
Parents: 39da22c 70ba4f0
Author: Luke Cwik 
Authored: Thu Jun 9 10:43:30 2016 -0700
Committer: Luke Cwik 
Committed: Thu Jun 9 10:43:30 2016 -0700

--
 pom.xml | 1 +
 1 file changed, 1 insertion(+)
--

[1/2] incubator-beam git commit: Disable exec-maven-plugin cleanupDaemonThreads

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 39da22c76 -> 983d467a7


Disable exec-maven-plugin cleanupDaemonThreads

exec-maven-plugin has a bad interaction with DataflowPipelineRunner
when waiting on daemon threads. When running example pipelines,
Maven will emit a warning saying background threads are alive
after being aborted. This looks to users like an error while there's
actually no harm to it. Disabling cleanupDaemonThreads suppresses
this warning.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/70ba4f0d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/70ba4f0d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/70ba4f0d

Branch: refs/heads/master
Commit: 70ba4f0dd63d564205b58d0ba4e5215f6802f8c9
Parents: 39da22c
Author: Scott Wegner 
Authored: Thu Jun 9 08:46:37 2016 -0700
Committer: Luke Cwik 
Committed: Thu Jun 9 10:43:22 2016 -0700

--
 pom.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/70ba4f0d/pom.xml
--
diff --git a/pom.xml b/pom.xml
index e7cc5f7..49b77fa 100644
--- a/pom.xml
+++ b/pom.xml
@@ -754,6 +754,7 @@
 
   
   
+false
 
   
 java.util.logging.config.file

[jira] [Commented] (BEAM-159) Support fixed number of shards in custom sinks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322846#comment-15322846
 ] 

ASF GitHub Bot commented on BEAM-159:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/436


> Support fixed number of shards in custom sinks
> --
>
> Key: BEAM-159
> URL: https://issues.apache.org/jira/browse/BEAM-159
> Project: Beam
>  Issue Type: Improvement
>Reporter: Eugene Kirpichov
>
> TextIO supports .withNumShards, however custom sinks, in particular 
> FileBasedSinks, provide no support for controlling sharding. Some users want 
> this, e.g. 
> http://stackoverflow.com/questions/36316304/set-num-of-output-shard-in-write-tosink-in-dataflow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #436: Raise error rather than overwrite data whe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/436


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: FileBasedSink: Detect bad shard name templates

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 61d8cf2c4 -> 39da22c76


FileBasedSink: Detect bad shard name templates

This is particularly relevant when TextIO.Write.withoutSharding() is
used: [BEAM-159].


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4d036bc8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4d036bc8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4d036bc8

Branch: refs/heads/master
Commit: 4d036bc8a4b8c255d09467d09fe4ed7eae9dd035
Parents: 61d8cf2
Author: Robert Bradshaw 
Authored: Wed Jun 8 14:59:56 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jun 9 09:47:24 2016 -0700

--
 .../org/apache/beam/sdk/io/FileBasedSink.java   | 16 +-
 .../apache/beam/sdk/io/FileBasedSinkTest.java   | 23 
 2 files changed, 33 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4d036bc8/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
index 9048380..521f54b 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
@@ -30,6 +30,7 @@ import org.apache.beam.sdk.util.IOChannelUtils;
 import org.apache.beam.sdk.util.MimeTypes;
 
 import com.google.common.base.Preconditions;
+import com.google.common.collect.Ordering;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -43,7 +44,7 @@ import java.nio.file.Paths;
 import java.nio.file.StandardCopyOption;
 import java.util.ArrayList;
 import java.util.Collection;
-import java.util.Collections;
+import java.util.HashSet;
 import java.util.List;
 
 /**
@@ -333,13 +334,10 @@ public abstract class FileBasedSink extends Sink {
 protected final List copyToOutputFiles(List filenames, 
PipelineOptions options)
 throws IOException {
   int numFiles = filenames.size();
-  List srcFilenames = new ArrayList<>();
+  // Sort files for idempotence.
+  List srcFilenames = Ordering.natural().sortedCopy(filenames);
   List destFilenames = generateDestinationFilenames(numFiles);
 
-  // Sort files for copying.
-  srcFilenames.addAll(filenames);
-  Collections.sort(srcFilenames);
-
   if (numFiles > 0) {
 LOG.debug("Copying {} files.", numFiles);
 FileOperations fileOperations =
@@ -366,6 +364,12 @@ public abstract class FileBasedSink extends Sink {
 destFilenames.add(IOChannelUtils.constructName(
 baseOutputFilename, fileNamingTemplate, suffix, i, numFiles));
   }
+
+  int numDistinctShards = new HashSet(destFilenames).size();
+  Preconditions.checkState(numDistinctShards == numFiles,
+  "Shard name template '%s' only generated %s distinct file names for 
%s files.",
+  fileNamingTemplate, numDistinctShards, numFiles);
+
   return destFilenames;
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4d036bc8/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
index 0e434fc..d3454da 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileBasedSinkTest.java
@@ -22,6 +22,7 @@ import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
 
 import org.apache.beam.sdk.io.FileBasedSink.FileBasedWriteOperation;
 import 
org.apache.beam.sdk.io.FileBasedSink.FileBasedWriteOperation.TemporaryFileRetention;
@@ -374,6 +375,28 @@ public class FileBasedSinkTest {
   }
 
   /**
+   * Reject non-distinct output filenames.
+   */
+  @Test
+  public void testCollidingOutputFilenames() {
+SimpleSink sink = new SimpleSink("output", "test", "-NN");
+SimpleSink.SimpleWriteOperation writeOp = new 
SimpleSink.SimpleWriteOperation(sink);
+
+// A single shard doesn't need to include the shard number.
+assertEquals(Arrays.asList("output-01.test"),
+ writeOp.generateDestinationFilenames(1));
+
+// More than one shard does.
+try {
+  writeOp.generateDestinationFilenames(3);
+  

[2/2] incubator-beam git commit: Closes #436

[dhalperi]

Closes #436


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/39da22c7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/39da22c7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/39da22c7

Branch: refs/heads/master
Commit: 39da22c7636bd93c6f97ccaa5cd918c6de5dcdae
Parents: 61d8cf2 4d036bc
Author: Dan Halperin 
Authored: Thu Jun 9 09:47:25 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jun 9 09:47:25 2016 -0700

--
 .../org/apache/beam/sdk/io/FileBasedSink.java   | 16 +-
 .../apache/beam/sdk/io/FileBasedSinkTest.java   | 23 
 2 files changed, 33 insertions(+), 6 deletions(-)
--

[jira] [Created] (BEAM-330) Maven exec warning message on Dataflow WordCount Example

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-330:
-

 Summary: Maven exec warning message on Dataflow WordCount Example
 Key: BEAM-330
 URL: https://issues.apache.org/jira/browse/BEAM-330
 Project: Beam
  Issue Type: Bug
  Components: examples-java
Reporter: Scott Wegner
Assignee: Scott Wegner
Priority: Minor


When running the WordCount example with DataflowPipelineRunner, sometimes the 
Maven process will emit a warning when trying to tear down the process:

{quote}{noformat}
[DEBUG] interrupting thread 
Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount]
[DEBUG] joining on thread 
Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount]
[WARNING] thread Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount] 
was interrupted but is still alive after waiting at least 14999msecs
[WARNING] thread Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount] 
will linger despite being asked to die via interruption
[WARNING] NOTE: 1 thread(s) did not finish despite being asked to  via 
interruption. This is not a problem with exec:java, it is a problem with the 
running code. Although not serious, it should be remedied.
[WARNING] Couldn't destroy threadgroup 
org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=org.apache.beam.examples.WordCount,maxpri=10]
{noformat}{quote}

This appears to be some bad interaction between exec-maven-plugin and 
DataflowPipelineRunner, possibly due to exec-maven-plugin's use of Guice and 
our shading of it.

The problem doesn't always reproduce, except when executing multiple Maven 
targets, such as "mvn install exec:java \[...\]"

Disabling exec:java's cleanupDaemonThreads indeed suppresses the issue.  I 
recommend we add this configuration to the root pom.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-330) Maven exec warning message on Dataflow WordCount Example

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322745#comment-15322745
 ] 

ASF GitHub Bot commented on BEAM-330:
-

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/439

[BEAM-330] Disable exec-maven-plugin cleanupDaemonThreads

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

exec-maven-plugin has a bad interaction with DataflowPipelineRunner
when waiting on daemon threads. When running example pipelines,
Maven will emit a warning saying background threads are alive
after being aborted. This looks to users like an error while there's
actually no harm to it. Disabling cleanupDaemonThreads suppresses
this warning.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam 
exec-cleanupDaemonThreads

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/439.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #439


commit 10154c6f0978cf6f33ae7153e85a02607b6f65ed
Author: Scott Wegner 
Date:   2016-06-09T15:46:37Z

Disable exec-maven-plugin cleanupDaemonThreads

exec-maven-plugin has a bad interaction with DataflowPipelineRunner
when waiting on daemon threads. When running example pipelines,
Maven will emit a warning saying background threads are alive
after being aborted. This looks to users like an error while there's
actually no harm to it. Disabling cleanupDaemonThreads suppresses
this warning.




> Maven exec warning message on Dataflow WordCount Example
> 
>
> Key: BEAM-330
> URL: https://issues.apache.org/jira/browse/BEAM-330
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> When running the WordCount example with DataflowPipelineRunner, sometimes the 
> Maven process will emit a warning when trying to tear down the process:
> {quote}{noformat}
> [DEBUG] interrupting thread 
> Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount]
> [DEBUG] joining on thread 
> Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount]
> [WARNING] thread Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount] 
> was interrupted but is still alive after waiting at least 14999msecs
> [WARNING] thread Thread[pool-1-thread-1,5,org.apache.beam.examples.WordCount] 
> will linger despite being asked to die via interruption
> [WARNING] NOTE: 1 thread(s) did not finish despite being asked to  via 
> interruption. This is not a problem with exec:java, it is a problem with the 
> running code. Although not serious, it should be remedied.
> [WARNING] Couldn't destroy threadgroup 
> org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=org.apache.beam.examples.WordCount,maxpri=10]
> {noformat}{quote}
> This appears to be some bad interaction between exec-maven-plugin and 
> DataflowPipelineRunner, possibly due to exec-maven-plugin's use of Guice and 
> our shading of it.
> The problem doesn't always reproduce, except when executing multiple Maven 
> targets, such as "mvn install exec:java \[...\]"
> Disabling exec:java's cleanupDaemonThreads indeed suppresses the issue.  I 
> recommend we add this configuration to the root pom.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #439: [BEAM-330] Disable exec-maven-plugin clean...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/439

[BEAM-330] Disable exec-maven-plugin cleanupDaemonThreads

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

exec-maven-plugin has a bad interaction with DataflowPipelineRunner
when waiting on daemon threads. When running example pipelines,
Maven will emit a warning saying background threads are alive
after being aborted. This looks to users like an error while there's
actually no harm to it. Disabling cleanupDaemonThreads suppresses
this warning.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam 
exec-cleanupDaemonThreads

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/439.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #439


commit 10154c6f0978cf6f33ae7153e85a02607b6f65ed
Author: Scott Wegner 
Date:   2016-06-09T15:46:37Z

Disable exec-maven-plugin cleanupDaemonThreads

exec-maven-plugin has a bad interaction with DataflowPipelineRunner
when waiting on daemon threads. When running example pipelines,
Maven will emit a warning saying background threads are alive
after being aborted. This looks to users like an error while there's
actually no harm to it. Disabling cleanupDaemonThreads suppresses
this warning.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-329) README cluster example not working with HDFS input

[Amit Sela (JIRA)]

Amit Sela created BEAM-329:
--

 Summary: README cluster example not working with HDFS input
 Key: BEAM-329
 URL: https://issues.apache.org/jira/browse/BEAM-329
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Amit Sela
Priority: Minor


In the Spark runner README - 
https://github.com/apache/incubator-beam/tree/master/runners/spark 
The "Running On Cluster" example doesn't work because it looks for the input on 
the local FS instead of HDFS.
Need to check what's up with TextIO there.. If this behaviour is wanted we 
should fix the README, otherwise it should read from HDFS. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-329) README cluster example not working with HDFS input

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-329:
---
Affects Version/s: 0.1.0-incubating

> README cluster example not working with HDFS input
> --
>
> Key: BEAM-329
> URL: https://issues.apache.org/jira/browse/BEAM-329
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.1.0-incubating
>Reporter: Amit Sela
>Priority: Minor
>
> In the Spark runner README - 
> https://github.com/apache/incubator-beam/tree/master/runners/spark 
> The "Running On Cluster" example doesn't work because it looks for the input 
> on the local FS instead of HDFS.
> Need to check what's up with TextIO there.. If this behaviour is wanted we 
> should fix the README, otherwise it should read from HDFS. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #509

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_MavenVerify #601

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Flink :: Core #601

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_MavenVerify #600

[Apache Jenkins Server]

See 

--
[...truncated 6122 lines...]
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
 (24 KB at 19.2 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils-bean-collections/1.8.3/commons-beanutils-bean-collections-1.8.3.jar
 (26 KB at 21.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/javax/inject/javax.inject/1/javax.inject-1.jar
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/aopalliance/aopalliance/1.0/aopalliance-1.0.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/javax/inject/javax.inject/1/javax.inject-1.jar
 (3 KB at 1.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-clients_2.10/1.0.0/flink-clients_2.10-1.0.0.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/aopalliance/aopalliance/1.0/aopalliance-1.0.jar
 (5 KB at 3.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.0.0/flink-optimizer_2.10-1.0.0.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/inject/guice/3.0/guice-3.0.jar 
(694 KB at 541.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.0.0/flink-streaming-java_2.10-1.0.0-tests.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.0.0/flink-streaming-java_2.10-1.0.0-tests.jar
 (632 KB at 458.3 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.0.0/flink-test-utils_2.10-1.0.0.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-clients_2.10/1.0.0/flink-clients_2.10-1.0.0.jar
 (2414 KB at 1513.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.0.0/flink-core-1.0.0-tests.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.0.0/flink-optimizer_2.10-1.0.0.jar
 (2683 KB at 1663.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.0.0/flink-runtime_2.10-1.0.0-tests.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.0.0/flink-test-utils_2.10-1.0.0.jar
 (2401 KB at 1472.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scalatest/scalatest_2.10/2.2.2/scalatest_2.10-2.2.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-core/1.0.0/flink-core-1.0.0-tests.jar
 (632 KB at 381.8 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.10.4/scala-reflect-2.10.4.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-runtime_2.10/1.0.0/flink-runtime_2.10-1.0.0-tests.jar
 (1884 KB at 1051.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-test/2.8.0/curator-test-2.8.0.jar
 (39 KB at 21.3 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-reflect/2.10.4/scala-reflect-2.10.4.jar
 (3129 KB at 1634.5 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop2/1.0.0/flink-shaded-hadoop2-1.0.0.jar
 (17893 KB at 8548.5 KB/sec)
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scalatest/scalatest_2.10/2.2.2/scalatest_2.10-2.2.2.jar
 (7326 KB at 3408.6 KB/sec)
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection timed out
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection timed out
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection timed out
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] 
[INFO] 
[INFO] Skipping Apache Beam :: Parent
[INFO] This project has been banned from the build due to previous failures.
[INFO] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [ 25.118 s]
[INFO] Apache Beam :: SDKs 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #508

[Apache Jenkins Server]

See