[GitHub] incubator-beam-site pull request #108: Exclude apex runner translation packa...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam-site/pull/108

Exclude apex runner translation package from the generated javadoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam-site 
RELEASE_GUIDE_UPDATE

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/108.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #108


commit c7ac43e62f01950614dc7cc1ec5137cb2bd7a8e6
Author: Jean-Baptiste Onofré 
Date:   2016-12-15T09:35:32Z

Update Release Guide: exclude apex runner translation package from the 
generated javadoc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #109: Publish 0.4.0-incubating release

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam-site/pull/109

Publish 0.4.0-incubating release



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam-site release-0.4.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/109.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #109


commit f0b73ef95a1e2c709cffd4a1402cc7441f2d24be
Author: Jean-Baptiste Onofré 
Date:   2016-12-15T10:14:08Z

Publish 0.4.0-incubating release




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #110: Update website pull request instructi...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam-site/pull/110

Update website pull request instructions in the Release Guide



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam-site 
RELEASE_GUIDE_WEBSITE_PR_UPDATE

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/110.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #110


commit 8ddeada95adafa0949da8313ff6ab1f780a3a493
Author: Jean-Baptiste Onofré 
Date:   2016-12-15T11:58:30Z

Update website pull request instructions in the Release Guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-855) Remove the need for --streaming option in the spark runner

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-855:
---
Fix Version/s: (was: 0.4.0-incubating)

> Remove the need for --streaming option in the spark runner
> --
>
> Key: BEAM-855
> URL: https://issues.apache.org/jira/browse/BEAM-855
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Ismaël Mejía
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-807) [SparkRunner] Replace OldDoFn with DoFn

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-807:
---
Fix Version/s: (was: 0.4.0-incubating)
   0.5.0-incubating

> [SparkRunner] Replace OldDoFn with DoFn
> ---
>
> Key: BEAM-807
> URL: https://issues.apache.org/jira/browse/BEAM-807
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.5.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-757) The SparkRunner should utilize the SDK's DoFnRunner instead of writing it's own.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-757:
---
Fix Version/s: (was: 0.4.0-incubating)
   0.5.0-incubating

> The SparkRunner should utilize the SDK's DoFnRunner instead of writing it's 
> own.
> 
>
> Key: BEAM-757
> URL: https://issues.apache.org/jira/browse/BEAM-757
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.5.0-incubating
>
>
> The SDK now provides DoFnRunner implementations, and so to avoid maintaining 
> against the SDK, the runner should leverage the runner API instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-855) Remove the need for --streaming option in the spark runner

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela closed BEAM-855.
--
   Resolution: Duplicate
Fix Version/s: Not applicable

> Remove the need for --streaming option in the spark runner
> --
>
> Key: BEAM-855
> URL: https://issues.apache.org/jira/browse/BEAM-855
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-855) Remove the need for --streaming option in the spark runner

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela reopened BEAM-855:

  Assignee: (was: Jean-Baptiste Onofré)

> Remove the need for --streaming option in the spark runner
> --
>
> Key: BEAM-855
> URL: https://issues.apache.org/jira/browse/BEAM-855
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Ismaël Mejía
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-932] Enable findbugs validation (and fix existing issues)

[amitsela]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 1ad638e51 -> 4323247a3


[BEAM-932] Enable findbugs validation (and fix existing issues)


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bba3700a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bba3700a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bba3700a

Branch: refs/heads/master
Commit: bba3700aebd9dcaf88402b6845008cf7e5950cfe
Parents: 1ad638e
Author: Ismaël Mejía 
Authored: Thu Dec 15 14:34:20 2016 +0100
Committer: Ismaël Mejía 
Committed: Thu Dec 15 14:34:20 2016 +0100

--
 runners/spark/pom.xml   | 10 
 .../metrics/WithNamedAggregatorsSupport.java|  4 +-
 .../spark/translation/WindowingHelpers.java |  3 +-
 .../src/main/resources/beam/findbugs-filter.xml | 49 +---
 4 files changed, 47 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bba3700a/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index e8fffa2..5a2fe87 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -146,7 +146,6 @@
 
   com.google.code.findbugs
   jsr305
-  1.3.9
 
 
   com.google.guava
@@ -317,15 +316,6 @@
   
 
   
-
-
-  org.codehaus.mojo
-  findbugs-maven-plugin
-  
-true
-  
-
-
 
   org.apache.maven.plugins
   maven-surefire-plugin

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bba3700a/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/metrics/WithNamedAggregatorsSupport.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/metrics/WithNamedAggregatorsSupport.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/metrics/WithNamedAggregatorsSupport.java
index 6932ae6..5e71280 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/metrics/WithNamedAggregatorsSupport.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/aggregators/metrics/WithNamedAggregatorsSupport.java
@@ -121,8 +121,8 @@ public class WithNamedAggregatorsSupport extends 
MetricRegistry {
 final String parentName = entry.getKey();
 final Map gaugeMap = 
Maps.transformEntries(agg.renderAll(), toGauge());
 final Map fullNameGaugeMap = Maps.newLinkedHashMap();
-for (String shortName : gaugeMap.keySet()) {
-  fullNameGaugeMap.put(parentName + "." + shortName, 
gaugeMap.get(shortName));
+for (Map.Entry gaugeEntry : gaugeMap.entrySet()) {
+  fullNameGaugeMap.put(parentName + "." + gaugeEntry.getKey(), 
gaugeEntry.getValue());
 }
 return Maps.filterValues(fullNameGaugeMap, Predicates.notNull());
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bba3700a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/WindowingHelpers.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/WindowingHelpers.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/WindowingHelpers.java
index ec94f3e..0acff71 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/WindowingHelpers.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/WindowingHelpers.java
@@ -18,6 +18,7 @@
 
 package org.apache.beam.runners.spark.translation;
 
+import javax.annotation.Nonnull;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.spark.api.java.function.Function;
 
@@ -84,7 +85,7 @@ public final class WindowingHelpers {
   public static  com.google.common.base.Function, T> 
unwindowValueFunction() {
 return new com.google.common.base.Function, T>() {
   @Override
-  public T apply(WindowedValue t) {
+  public T apply(@Nonnull WindowedValue t) {
 return t.getValue();
   }
 };

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bba3700a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml 
b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
index a696aeb..bfb4988 100644
--- a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
@@ -125,6 +1

[2/2] incubator-beam git commit: This closes #1463

[amitsela]

This closes #1463


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4323247a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4323247a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4323247a

Branch: refs/heads/master
Commit: 4323247a3a8ea6fc06b99f66f0001f6956b494c9
Parents: 1ad638e bba3700
Author: Sela 
Authored: Thu Dec 15 16:16:57 2016 +0200
Committer: Sela 
Committed: Thu Dec 15 16:16:57 2016 +0200

--
 runners/spark/pom.xml   | 10 
 .../metrics/WithNamedAggregatorsSupport.java|  4 +-
 .../spark/translation/WindowingHelpers.java |  3 +-
 .../src/main/resources/beam/findbugs-filter.xml | 49 +---
 4 files changed, 47 insertions(+), 19 deletions(-)
--

[GitHub] incubator-beam pull request #1463: [BEAM-932] Enable findbugs validation (an...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1463


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-932) Findbugs doesn't pass in Spark runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15751544#comment-15751544
 ] 

ASF GitHub Bot commented on BEAM-932:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1463


> Findbugs doesn't pass in Spark runner
> -
>
> Key: BEAM-932
> URL: https://issues.apache.org/jira/browse/BEAM-932
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Daniel Halperin
>Assignee: Ismaël Mejía
>
> {code}
> [INFO] --- findbugs-maven-plugin:3.0.1:check (default) @ beam-runners-spark 
> ---
> [INFO] BugInstance size is 19
> [INFO] Error size is 0
> [INFO] Total bugs: 19
> [INFO] instanceof will always return false in 
> org.apache.beam.runners.spark.SparkRunner.run(Pipeline), since a 
> RuntimeException can't be a org.apache.spark.SparkException 
> [org.apache.beam.runners.spark.SparkRunner] At SparkRunner.java:[line 161]
> [INFO] 
> org.apache.beam.runners.spark.aggregators.metrics.WithNamedAggregatorsSupport$1.apply(Map$Entry)
>  makes inefficient use of keySet iterator instead of entrySet iterator 
> [org.apache.beam.runners.spark.aggregators.metrics.WithNamedAggregatorsSupport$1]
>  At WithNamedAggregatorsSupport.java:[line 125]
> [INFO] The class name 
> org.apache.beam.runners.spark.aggregators.metrics.sink.CsvSink shadows the 
> simple name of the superclass org.apache.spark.metrics.sink.CsvSink 
> [org.apache.beam.runners.spark.aggregators.metrics.sink.CsvSink] At 
> CsvSink.java:[lines 37-38]
> [INFO] The class name 
> org.apache.beam.runners.spark.aggregators.metrics.sink.GraphiteSink shadows 
> the simple name of the superclass org.apache.spark.metrics.sink.GraphiteSink 
> [org.apache.beam.runners.spark.aggregators.metrics.sink.GraphiteSink] At 
> GraphiteSink.java:[lines 37-38]
> [INFO] t must be non-null but is marked as nullable 
> [org.apache.beam.runners.spark.translation.WindowingHelpers$4] At 
> WindowingHelpers.java:[line 88]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$1.evaluate(ConsoleIO$Write$Unbound,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$1]
>  At StreamingTransformTranslator.java:[line 92]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$10.evaluate(ParDo$Bound,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$10]
>  At StreamingTransformTranslator.java:[line 360]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$11.evaluate(ParDo$BoundMulti,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$11]
>  At StreamingTransformTranslator.java:[line 395]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$2.evaluate(Read$Unbounded,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$2]
>  At StreamingTransformTranslator.java:[line 104]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$3.evaluate(CreateStream$QueuedValues,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$3]
>  At StreamingTransformTranslator.java:[line 115]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$4.evaluate(Flatten$FlattenPCollectionList,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$4]
>  At StreamingTransformTran

[jira] [Resolved] (BEAM-932) Findbugs doesn't pass in Spark runner

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-932.

   Resolution: Fixed
Fix Version/s: 0.5.0-incubating

> Findbugs doesn't pass in Spark runner
> -
>
> Key: BEAM-932
> URL: https://issues.apache.org/jira/browse/BEAM-932
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Daniel Halperin
>Assignee: Ismaël Mejía
> Fix For: 0.5.0-incubating
>
>
> {code}
> [INFO] --- findbugs-maven-plugin:3.0.1:check (default) @ beam-runners-spark 
> ---
> [INFO] BugInstance size is 19
> [INFO] Error size is 0
> [INFO] Total bugs: 19
> [INFO] instanceof will always return false in 
> org.apache.beam.runners.spark.SparkRunner.run(Pipeline), since a 
> RuntimeException can't be a org.apache.spark.SparkException 
> [org.apache.beam.runners.spark.SparkRunner] At SparkRunner.java:[line 161]
> [INFO] 
> org.apache.beam.runners.spark.aggregators.metrics.WithNamedAggregatorsSupport$1.apply(Map$Entry)
>  makes inefficient use of keySet iterator instead of entrySet iterator 
> [org.apache.beam.runners.spark.aggregators.metrics.WithNamedAggregatorsSupport$1]
>  At WithNamedAggregatorsSupport.java:[line 125]
> [INFO] The class name 
> org.apache.beam.runners.spark.aggregators.metrics.sink.CsvSink shadows the 
> simple name of the superclass org.apache.spark.metrics.sink.CsvSink 
> [org.apache.beam.runners.spark.aggregators.metrics.sink.CsvSink] At 
> CsvSink.java:[lines 37-38]
> [INFO] The class name 
> org.apache.beam.runners.spark.aggregators.metrics.sink.GraphiteSink shadows 
> the simple name of the superclass org.apache.spark.metrics.sink.GraphiteSink 
> [org.apache.beam.runners.spark.aggregators.metrics.sink.GraphiteSink] At 
> GraphiteSink.java:[lines 37-38]
> [INFO] t must be non-null but is marked as nullable 
> [org.apache.beam.runners.spark.translation.WindowingHelpers$4] At 
> WindowingHelpers.java:[line 88]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$1.evaluate(ConsoleIO$Write$Unbound,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$1]
>  At StreamingTransformTranslator.java:[line 92]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$10.evaluate(ParDo$Bound,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$10]
>  At StreamingTransformTranslator.java:[line 360]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$11.evaluate(ParDo$BoundMulti,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$11]
>  At StreamingTransformTranslator.java:[line 395]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$2.evaluate(Read$Unbounded,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$2]
>  At StreamingTransformTranslator.java:[line 104]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$3.evaluate(CreateStream$QueuedValues,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$3]
>  At StreamingTransformTranslator.java:[line 115]
> [INFO] Unchecked/unconfirmed cast from 
> org.apache.beam.runners.spark.translation.EvaluationContext to 
> org.apache.beam.runners.spark.translation.streaming.StreamingEvaluationContext
>  in 
> org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$4.evaluate(Flatten$FlattenPCollectionList,
>  EvaluationContext) 
> [org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$4]
>  At StreamingTransformTranslator.java:[line 128]
> [INFO] Unchecked/unconfirmed cast from 
> 

[jira] [Commented] (BEAM-690) Backoff in the DirectRunner Monitor if no work is Available

[Stas Levin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15751670#comment-15751670
 ] 

Stas Levin commented on BEAM-690:
-

Hi,

I'm trying to look into it, correct me if I'm wrong, but won't having such a 
back-off result in tests taking longer to complete? 

Backing-off the scheduling of {{MonitorRunnable#run}} delays the detection of a 
shut down performed in {{MonitorRunnable#shouldShutdown}}, called by 
{{MonitorRunnable#run}}.

> Backoff in the DirectRunner Monitor if no work is Available
> ---
>
> Key: BEAM-690
> URL: https://issues.apache.org/jira/browse/BEAM-690
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>
> When a Pipeline has no elements available to process, the Monitor Runnable 
> will be repeatedly scheduled. Given that there is no work to be done, this 
> will loop over the steps in the transform looking for timers, and prompt the 
> sources to perform additional work, even though there is no work to be done. 
> This consumes the entirety of a single core.
> Add a bounded backoff to rescheduling the monitor runnable if no work has 
> been done since it last ran. This will reduce resource consumption on 
> low-throughput Pipelines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1162) Create RedisIO

[JIRA]

Ismaël Mejía created BEAM-1162:
--

 Summary: Create RedisIO
 Key: BEAM-1162
 URL: https://issues.apache.org/jira/browse/BEAM-1162
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Affects Versions: Not applicable
Reporter: Ismaël Mejía
Assignee: Jean-Baptiste Onofré
Priority: Minor


As discussed in the mailing list there is interest in this IO.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1163) Add signature keys to the release guide vote template

[JIRA]

Ismaël Mejía created BEAM-1163:
--

 Summary: Add signature keys to the release guide vote template
 Key: BEAM-1163
 URL: https://issues.apache.org/jira/browse/BEAM-1163
 Project: Beam
  Issue Type: Improvement
  Components: website
Affects Versions: Not applicable
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía
Priority: Trivial


A small improvement, the idea is to add just the fingerprint of the person who 
signed the release in the template (for validation purposes:

The release artifacts are signed with the key with fingerprint XXX
https://dist.apache.org/repos/dist/release/incubator/beam/KEYS




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Apex #1

[Apache Jenkins Server]

See 


--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/incubator-beam.git
 > git init 
 > 
 >  # timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4323247a3a8ea6fc06b99f66f0001f6956b494c9 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4323247a3a8ea6fc06b99f66f0001f6956b494c9
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Discovered a new module org.apache.beam:beam-parent Apache Beam :: Parent
Discovered a new module org.apache.beam:beam-sdks-java-build-tools Apache Beam 
:: SDKs :: Java :: Build Tools
Discovered a new module org.apache.beam:beam-sdks-parent Apache Beam :: SDKs
Discovered a new module org.apache.beam:beam-sdks-java-parent Apache Beam :: 
SDKs :: Java
Discovered a new module org.apache.beam:beam-sdks-java-core Apache Beam :: SDKs 
:: Java :: Core
Discovered a new module org.apache.beam:beam-sdks-java-io-parent Apache Beam :: 
SDKs :: Java :: IO
Discovered a new module org.apache.beam:beam-sdks-java-io-google-cloud-platform 
Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform
Discovered a new module org.apache.beam:beam-sdks-java-io-hdfs Apache Beam :: 
SDKs :: Java :: IO :: HDFS
Discovered a new module org.apache.beam:beam-sdks-java-io-jms Apache Beam :: 
SDKs :: Java :: IO :: JMS
Discovered a new module org.apache.beam:beam-sdks-java-io-kafka Apache Beam :: 
SDKs :: Java :: IO :: Kafka
Discovered a new module org.apache.beam:beam-sdks-java-io-kinesis Apache Beam 
:: SDKs :: Java :: IO :: Kinesis
Discovered a new module org.apache.beam:beam-sdks-java-io-mongodb Apache Beam 
:: SDKs :: Java :: IO :: MongoDB
Discovered a new module org.apache.beam:beam-sdks-java-io-jdbc Apache Beam :: 
SDKs :: Java :: IO :: JDBC
Discovered a new module org.apache.beam:beam-sdks-java-maven-archetypes-parent 
Apache Beam :: SDKs :: Java :: Maven Archetypes
Discovered a new module org.apache.beam:beam-sdks-java-maven-archetypes-starter 
Apache Beam :: SDKs :: Java :: Maven Archetypes :: Starter
Discovered a new module 
org.apache.beam:beam-sdks-java-maven-archetypes-examples Apache Beam :: SDKs :: 
Java :: Maven Archetypes :: Examples
Discovered a new module 
org.apache.beam:beam-sdks-java-maven-archetypes-examples-java8 Apache Beam :: 
SDKs :: Java :: Maven Archetypes :: Examples - Java 8
Discovered a new module org.apache.beam:beam-sdks-java-extensions-parent Apache 
Beam :: SDKs :: Java :: Extensions
Discovered a new module org.apache.beam:beam-sdks-java-extensions-join-library 
Apache Beam :: SDKs :: Java :: Extensions :: Join library
Discovered a new module org.apache.beam:beam-sdks-java-extensions-sorter Apache 
Beam :: SDKs :: Java :: Extensions :: Sorter
Discovered a new module org.apache.beam:beam-runners-parent Apache Beam :: 
Runners
Discovered a new module org.apache.beam:beam-runners-core-java Apache Beam :: 
Runners :: Core Java
Discovered a new module org.apache.beam:beam-runners-direct-java Apache Beam :: 
Runners :: Direct Java
Discovered a new module org.apache.beam:beam-runners-flink-parent Apache Beam 
:: Runners :: Flink
Discovered a new module org.apache.beam:beam-runners-flink_2.10 Apache Beam :: 
Runners :: Flink :: Core
Discovered a new module org.apache.beam:beam-runners-flink_2.10-examples Apache 
Beam :: Runners :: Flink :: Examples
Discovered

[jira] [Assigned] (BEAM-647) Fault-tolerant sideInputs via Broadcast variables

[Kobi Salant (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kobi Salant reassigned BEAM-647:


Assignee: Kobi Salant  (was: Amit Sela)

> Fault-tolerant sideInputs via Broadcast variables
> -
>
> Key: BEAM-647
> URL: https://issues.apache.org/jira/browse/BEAM-647
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Kobi Salant
>
> Following https://github.com/apache/incubator-beam/pull/909 which enables 
> checkpointing to recover from failures, sideInputs (being implemented by 
> broadcast variables) should be handled in a specific manner as described 
> here: 
> http://spark.apache.org/docs/1.6.2/streaming-programming-guide.html#accumulators-and-broadcast-variables.
> This is a bit more complicated than Aggregators (via Accumulators) as they 
> are implemented using a single "aggregating"  Accumulator, while a pipeline 
> may contain multiple sideInputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-647) Fault-tolerant sideInputs via Broadcast variables

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752015#comment-15752015
 ] 

ASF GitHub Bot commented on BEAM-647:
-

GitHub user kobisalant opened a pull request:

https://github.com/apache/incubator-beam/pull/1624

[BEAM-647] Fault-tolerant sideInputs via Broadcast variables

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kobisalant/incubator-beam 
BEAM-647-Fault-tolerant-sideInputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1624


commit 228c613c70c4edf1b01294a487d49f0a5492136e
Author: ksalant 
Date:   2016-12-15T17:42:47Z

[BEAM-647] Fault-tolerant sideInputs via Broadcast variables




> Fault-tolerant sideInputs via Broadcast variables
> -
>
> Key: BEAM-647
> URL: https://issues.apache.org/jira/browse/BEAM-647
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Kobi Salant
>
> Following https://github.com/apache/incubator-beam/pull/909 which enables 
> checkpointing to recover from failures, sideInputs (being implemented by 
> broadcast variables) should be handled in a specific manner as described 
> here: 
> http://spark.apache.org/docs/1.6.2/streaming-programming-guide.html#accumulators-and-broadcast-variables.
> This is a bit more complicated than Aggregators (via Accumulators) as they 
> are implemented using a single "aggregating"  Accumulator, while a pipeline 
> may contain multiple sideInputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1624: [BEAM-647] Fault-tolerant sideInputs via ...

[kobisalant]

GitHub user kobisalant opened a pull request:

https://github.com/apache/incubator-beam/pull/1624

[BEAM-647] Fault-tolerant sideInputs via Broadcast variables

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kobisalant/incubator-beam 
BEAM-647-Fault-tolerant-sideInputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1624.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1624


commit 228c613c70c4edf1b01294a487d49f0a5492136e
Author: ksalant 
Date:   2016-12-15T17:42:47Z

[BEAM-647] Fault-tolerant sideInputs via Broadcast variables




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Apex #2

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/incubator-beam.git
 > git init 
 > 
 >  # timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 4323247a3a8ea6fc06b99f66f0001f6956b494c9 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 4323247a3a8ea6fc06b99f66f0001f6956b494c9
 > git rev-list 4323247a3a8ea6fc06b99f66f0001f6956b494c9 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Established TCP socket on 32896
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Apex] $ 
/home/jenkins/tools/java/latest1.8/bin/java -Xmx2g -Xms256m 
-XX:MaxPermSize=512m -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 32896
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 clean verify --projects runners/apex --also-make --batch-mode --errors 
--profile runnable-on-service-tests --profile local-runnable-on-service-tests
Unable to parse command line options: Unrecognized option: --profile

usage: mvn [options] [] []

Options:
 -am,--also-makeIf project list is specified, also
build projects required by the
list
 -amd,--also-make-dependentsIf project list is specified, also
build projects that depend on
projects on the list
 -B,--batch-modeRun in non-interactive (batch)
mode
 -b,--builder  The id of the build strategy to
use.
 -C,--strict-checksums  Fail the build if checksums don't
match
 -c,--lax-checksums Warn if checksums don't match
 -cpu,--check-plugin-updatesIneffective, only kept for
backward compatibility
 -D,--define   Define a system property
 -e,--errorsProduce execution error messages
 -emp,--encrypt-master-passwordEncrypt master security password
 -ep,--encrypt-passwordEncrypt server password
 -f,--file Force the use of an alternate POM
file (or directory with pom.xml).
 -fae,--fail-at-end Only fail the build afterwards;
allow all non-impacted builds to
   

[GitHub] incubator-beam pull request #1625: Don't incorrectly log error in MetricsEnv...

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/incubator-beam/pull/1625

Don't incorrectly log error in MetricsEnvironment

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [*] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [*] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [*] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Using getCurrentContainer() logs an error if metrics are not supported.
This is because it acts as the common point of access for user code that
reports metrics.

It should not be used within setCurrentContainer(), because the first
container being set will have a null previous-current-container, which
will cause the error to be incorrectly logged.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/incubator-beam fix-metrics-warning

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1625.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1625


commit 125223f8feb3576d3ff5ccdffa58a5e80808286c
Author: bchambers 
Date:   2016-12-14T19:23:39Z

Don't incorrectly log error in MetricsEnvironment

Using getCurrentContainer() logs an error if metrics are not supported.
This is because it acts as the common point of access for user code that
reports metrics.

It should not be used within setCurrentContainer(), because the first
container being set will have a null previous-current-container, which
will cause the error to be incorrectly logged.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1626: Fix typo in mvn command args in Apex post...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1626

Fix typo in mvn command args in Apex postcommit Jenkins job

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: anyone

Tested that this actually runs so kicking the seed job should fix up the 
postcommit for Apex.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam apex-mvn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1626.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1626


commit 8c00f63ca264e73c7ab7cfc760ef7748edbb6f60
Author: Kenneth Knowles 
Date:   2016-12-15T18:20:03Z

Fix mvn command args in Apex postcommit Jenkins job




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1627: [BEAM-1160] Add option to disable failure...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1627

[BEAM-1160] Add option to disable failures if filePattern resolves to empty

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Most PTransforms which take a filePattern have construction-time
validation which checks-- among other things-- that the specified
filePattern matches at least one file. This is particularly useful
for catching typos when specifying input files.

Most PTransforms also have an option to disable their
construction-time validation.  This is generally used when
validation cannot be performed at construction time: for example
because the proper credentials aren't available or the input
specification is late-bound in a template. To allow for these
scenarios and still guard against typos, FileBasedSource also
validates that the filePattern matches at least one file at
runtime.

This change adds the ability FileBasedSource to disable this runtime
validation, for cases uses case where empty filePatterns should be
allowed. FileBasedSource gains a new constructor parameter, and
PTransforms which use FileBasedSource have the option exposed in their
respective builder APIs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam read-allow-empty-glob

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1627.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1627


commit 43195ba27b4303c852e81fa1de493ec4fec641bb
Author: Scott Wegner 
Date:   2016-12-15T18:12:25Z

Add option to disable failures if filePattern resolves to empty

Most PTransforms which take a filePattern have construction-time
validation which checks-- among other things-- that the specified
filePattern matches at least one file. This is particularly useful
for catching typos when specifying input files.

Most PTransforms also have an option to disable their
construction-time validation.  This is generally used when
validation cannot be performed at construction time: for example
because the proper credentials aren't available or the input
specification is late-bound in a template. To allow for these
scenarios and still guard against typos, FileBasedSource also
validates that the filePattern matches at least one file at
runtime.

This change adds the ability FileBasedSource to disable this runtime
validation, for cases uses case where empty filePatterns should be
allowed. FileBasedSource gains a new constructor parameter, and
PTransforms which use FileBasedSource have the option exposed in their
respective builder APIs.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1160) Disabling Read transform validation cause empty file patterns to unexpected succeed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752113#comment-15752113
 ] 

ASF GitHub Bot commented on BEAM-1160:
--

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1627

[BEAM-1160] Add option to disable failures if filePattern resolves to empty

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Most PTransforms which take a filePattern have construction-time
validation which checks-- among other things-- that the specified
filePattern matches at least one file. This is particularly useful
for catching typos when specifying input files.

Most PTransforms also have an option to disable their
construction-time validation.  This is generally used when
validation cannot be performed at construction time: for example
because the proper credentials aren't available or the input
specification is late-bound in a template. To allow for these
scenarios and still guard against typos, FileBasedSource also
validates that the filePattern matches at least one file at
runtime.

This change adds the ability FileBasedSource to disable this runtime
validation, for cases uses case where empty filePatterns should be
allowed. FileBasedSource gains a new constructor parameter, and
PTransforms which use FileBasedSource have the option exposed in their
respective builder APIs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam read-allow-empty-glob

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1627.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1627


commit 43195ba27b4303c852e81fa1de493ec4fec641bb
Author: Scott Wegner 
Date:   2016-12-15T18:12:25Z

Add option to disable failures if filePattern resolves to empty

Most PTransforms which take a filePattern have construction-time
validation which checks-- among other things-- that the specified
filePattern matches at least one file. This is particularly useful
for catching typos when specifying input files.

Most PTransforms also have an option to disable their
construction-time validation.  This is generally used when
validation cannot be performed at construction time: for example
because the proper credentials aren't available or the input
specification is late-bound in a template. To allow for these
scenarios and still guard against typos, FileBasedSource also
validates that the filePattern matches at least one file at
runtime.

This change adds the ability FileBasedSource to disable this runtime
validation, for cases uses case where empty filePatterns should be
allowed. FileBasedSource gains a new constructor parameter, and
PTransforms which use FileBasedSource have the option exposed in their
respective builder APIs.




> Disabling Read transform validation cause empty file patterns to unexpected 
> succeed
> ---
>
> Key: BEAM-1160
> URL: https://issues.apache.org/jira/browse/BEAM-1160
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>
> Typically, input file patterns are validated during Pipeline construction, 
> but standard Read transforms include an option to disable validation. This is 
> generally useful but can lead to cases where a Pipeline executes successfully 
> with empty inputs.
> We should fail execution on empty file-based inputs even when validation is 
> disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1626

[kenn]

This closes #1626


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b38c9e9e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b38c9e9e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b38c9e9e

Branch: refs/heads/master
Commit: b38c9e9ebc38ed4d39c18ab1e24256111c298a11
Parents: 4323247 8c00f63
Author: Kenneth Knowles 
Authored: Thu Dec 15 10:34:48 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 10:34:48 2016 -0800

--
 .jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[1/2] incubator-beam git commit: Fix mvn command args in Apex postcommit Jenkins job

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4323247a3 -> b38c9e9eb


Fix mvn command args in Apex postcommit Jenkins job


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8c00f63c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8c00f63c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8c00f63c

Branch: refs/heads/master
Commit: 8c00f63ca264e73c7ab7cfc760ef7748edbb6f60
Parents: 4323247
Author: Kenneth Knowles 
Authored: Thu Dec 15 10:20:03 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 10:20:03 2016 -0800

--
 .jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8c00f63c/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
--
diff --git a/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy 
b/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
index 79d12be..232c94e 100644
--- a/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
+++ b/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
@@ -36,6 +36,6 @@ mavenJob('beam_PostCommit_Java_RunnableOnService_Apex') {
   --also-make \
   --batch-mode \
   --errors \
-  --profile runnable-on-service-tests \
-  --profile local-runnable-on-service-tests''')
+  --activate-profiles runnable-on-service-tests \
+  --activate-profiles local-runnable-on-service-tests''')
 }

[GitHub] incubator-beam pull request #1626: Fix typo in mvn command args in Apex post...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1626


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1628: Bump 0.4.0 incubating worker

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1628

Bump 0.4.0 incubating worker

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
bump-0.4.0-incubating-worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1628.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1628


commit 2c1d4eb4d878d1a8b8adf68b10eb8744cf524b5c
Author: Davor Bonaci 
Date:   2016-12-13T21:48:26Z

Update Maven Archetype versions to pin them to 0.4.0-incubating to be 
released

commit 9f3b0635971a45f8baf8fa68ba5ec2f8c47cec6b
Author: Davor Bonaci 
Date:   2016-12-13T22:18:32Z

Update Maven Archetype version; fixup for Java8.

commit 8d89bfce251d27c7fc6ff522a2cfeb676ec2c633
Author: Eugene Kirpichov 
Date:   2016-12-13T22:35:33Z

[BEAM-1149] Explode windows when fn uses side inputs

commit b2780881a6d735420a2d29c3270b781ea8a3b74f
Author: Kenneth Knowles 
Date:   2016-12-13T23:47:41Z

This closes #1602

commit f78d96069b428356609f219357cbf0702ec56c26
Author: Kenneth Knowles 
Date:   2016-12-14T00:09:57Z

Re-exclude UsesStatefulParDo tests for Dataflow

commit c1af44fa27633fd2a9592a13579415f6b974cfe6
Author: Kenneth Knowles 
Date:   2016-12-14T00:36:42Z

Revert "Allow stateful DoFn in DataflowRunner"

This reverts commit 42bb15d2df28b99b6788010450f41f2932095771.

The Dataflow service has introduced a bug that was masked by various
test disabling.

commit 10bb4767a1f989a1a75778828c07d9c72c450495
Author: Kenneth Knowles 
Date:   2016-12-14T01:08:42Z

This closes #1607

commit 122f05131c803ecaa082dfbc9ad6876b0ca467b8
Author: Kenneth Knowles 
Date:   2016-12-14T19:26:27Z

SimpleDoFnRunner observes window if SideInputReader is nonempty

commit d9f24b86c644ea85fd197eaab4c2d16b20a70d5f
Author: Kenneth Knowles 
Date:   2016-12-14T21:12:43Z

Fix NPE in StatefulParDoEvaluatorFactoryTest mocking

commit ea0d7f5c26fd97e8528d0614cdce8215ea05557c
Author: Kenneth Knowles 
Date:   2016-12-14T22:33:04Z

This closes #1619

commit af3989b086ed2e1bf0a08358ce6b7620c334cf49
Author: Davor Bonaci 
Date:   2016-12-15T02:58:57Z

[maven-release-plugin] prepare release v0.4.0-incubating-RC1

commit 7e234aa7938606d1a01064a6fdccf01bb90d4f2d
Author: Davor Bonaci 
Date:   2016-12-15T02:59:29Z

[maven-release-plugin] rollback changes from release preparation of 
v0.4.0-incubating-RC1

commit 21507b8da0bb74cc8d844ede985648ac4508df2b
Author: Kenneth Knowles 
Date:   2016-12-15T18:17:49Z

Set Dataflow worker version to 0.4.0-incubating




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1628: Bump 0.4.0 incubating worker

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1628


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1629: Set Dataflow worker version to 0.4.0-incu...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1629

Set Dataflow worker version to 0.4.0-incubating

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
bump-0.4.0-incubating-worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1629.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1629


commit 21507b8da0bb74cc8d844ede985648ac4508df2b
Author: Kenneth Knowles 
Date:   2016-12-15T18:17:49Z

Set Dataflow worker version to 0.4.0-incubating




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Set Dataflow worker version to 0.4.0-incubating

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating 7e234aa79 -> 5dcffe066


Set Dataflow worker version to 0.4.0-incubating


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/21507b8d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/21507b8d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/21507b8d

Branch: refs/heads/release-0.4.0-incubating
Commit: 21507b8da0bb74cc8d844ede985648ac4508df2b
Parents: 7e234aa
Author: Kenneth Knowles 
Authored: Thu Dec 15 10:17:49 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 10:17:49 2016 -0800

--
 .../org/apache/beam/runners/dataflow/dataflow.properties | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/21507b8d/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
 
b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
index 1eae8cb..abfea0c 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
+++ 
b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
@@ -18,6 +18,6 @@
 
 environment.major.version=6
 
-worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20161205
+worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-0.4.0-incubating
 
-worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20161205
+worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-0.4.0-incubating

[2/2] incubator-beam git commit: This closes #1629

[davor]

This closes #1629


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5dcffe06
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5dcffe06
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5dcffe06

Branch: refs/heads/release-0.4.0-incubating
Commit: 5dcffe066944e8cc302e716337770551efa48aed
Parents: 7e234aa 21507b8
Author: Davor Bonaci 
Authored: Thu Dec 15 10:50:08 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Dec 15 10:50:08 2016 -0800

--
 .../org/apache/beam/runners/dataflow/dataflow.properties | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[GitHub] incubator-beam pull request #1629: Set Dataflow worker version to 0.4.0-incu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1629


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1164) Allow a DoFn to opt in to mutating it's input

[Frances Perry (JIRA)]

Frances Perry created BEAM-1164:
---

 Summary: Allow a DoFn to opt in to mutating it's input
 Key: BEAM-1164
 URL: https://issues.apache.org/jira/browse/BEAM-1164
 Project: Beam
  Issue Type: Bug
  Components: beam-model
Reporter: Frances Perry
Priority: Minor


Runners generally can't tell if a DoFn is mutating inputs, but assuming so by 
default leads to significant performance implications from unnecessary copying 
(around sibling fusion, etc). So instead the model prevents mutating inputs, 
and the Direct Runner validates this behavior. (See: 
http://beam.incubator.apache.org/contribute/design-principles/#make-efficient-things-easy-rather-than-make-easy-things-efficient)
 

However, if users are processing a small number of large records by making 
incremental changes (for example, genomics use cases), the cost of immutability 
requirement can be very large. As a workaround, users sometimes do suboptimal 
things (fusing ParDos by hand) or undefined things when they expect the 
immutability requirement is unnecessarily strict (adding no-op coders in places 
they hope the runner won't be materializing things, mutating things anyway when 
they don't expect sibling fusion to happen, etc).

We should consider adding a signal (MutatingDoFn?) that users explicitly opt in 
to to say their code may mutate inputs. The runner can then use this assumption 
to either prevent optimizations that would break in the face of this or insert 
additional copies as needed to allow optimizations to preserve semantics.

See this related user@ discussion:
https://lists.apache.org/thread.html/f39689f54147117f3fc54c498eff1a20fa73f1be5b5cad5b6f816fd3@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1630: Cherry pick #1620 to release-0.4.0-incuba...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1630

Cherry pick #1620 to release-0.4.0-incubating

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The tests on #1620 have not come back yet, so please wait to merge until 
that is confirmed and merged on `master`. Just starting this PR to have 
something to reference.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
cherry-pick-CombineFnWithSideInputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1630.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1630


commit 7294f227b7b9ff660bcf8dbbb196662820438e36
Author: Eugene Kirpichov 
Date:   2016-12-14T22:29:30Z

[BEAM-1154] Get side input from proper window in ReduceFn




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1625

[tgroh]

This closes #1625


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c09fbd43
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c09fbd43
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c09fbd43

Branch: refs/heads/release-0.4.0-incubating
Commit: c09fbd43b0cdaa57cffc1925c313dcf91390adc7
Parents: 5dcffe0 e864ac2
Author: Thomas Groh 
Authored: Thu Dec 15 11:12:01 2016 -0800
Committer: Thomas Groh 
Committed: Thu Dec 15 11:12:01 2016 -0800

--
 .../main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Don't incorrectly log error in MetricsEnvironment

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating 5dcffe066 -> c09fbd43b


Don't incorrectly log error in MetricsEnvironment

Using getCurrentContainer() logs an error if metrics are not supported.
This is because it acts as the common point of access for user code that
reports metrics.

It should not be used within setCurrentContainer(), because the first
container being set will have a null previous-current-container, which
will cause the error to be incorrectly logged.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e864ac21
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e864ac21
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e864ac21

Branch: refs/heads/release-0.4.0-incubating
Commit: e864ac2141e31a930b85264506d06579ba6811de
Parents: 5dcffe0
Author: bchambers 
Authored: Wed Dec 14 11:23:39 2016 -0800
Committer: Thomas Groh 
Committed: Thu Dec 15 11:12:00 2016 -0800

--
 .../main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e864ac21/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
index 7c06cbf..5d7cb0b 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
@@ -56,7 +56,7 @@ public class MetricsEnvironment {
*/
   @Nullable
   public static MetricsContainer setCurrentContainer(@Nullable 
MetricsContainer container) {
-MetricsContainer previous = getCurrentContainer();
+MetricsContainer previous = CONTAINER_FOR_THREAD.get();
 if (container == null) {
   CONTAINER_FOR_THREAD.remove();
 } else {

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Apex #3

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1631: Change counter name in TestDataflowRunner

[joshualitt]

GitHub user joshualitt opened a pull request:

https://github.com/apache/incubator-beam/pull/1631

Change counter name in TestDataflowRunner

This small change updates the name of the windmill watermark counter in 
TestDataflowRunner



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/joshualitt/incubator-beam update_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1631.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1631


commit 2691a5878b00f2b0c313a4fb6d8d2c69d132effe
Author: Joshua Litt 
Date:   2016-12-15T19:07:27Z

Change counter name in TestDataflowRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1620

[kenn]

This closes #1620


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3c4b6930
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3c4b6930
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3c4b6930

Branch: refs/heads/master
Commit: 3c4b6930e1e934ab41052796524ef68887cd7915
Parents: b38c9e9 de109d5
Author: Kenneth Knowles 
Authored: Thu Dec 15 11:40:59 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 11:40:59 2016 -0800

--
 .../runners/core/ReduceFnContextFactory.java|  16 +--
 .../beam/runners/core/ReduceFnRunnerTest.java   | 133 ++-
 .../beam/sdk/transforms/CombineWithContext.java |   2 +-
 3 files changed, 78 insertions(+), 73 deletions(-)
--

[GitHub] incubator-beam pull request #1620: [BEAM-1154] Get side input from proper wi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1620


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-1154] Get side input from proper window in ReduceFn

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master b38c9e9eb -> 3c4b6930e


[BEAM-1154] Get side input from proper window in ReduceFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/de109d5b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/de109d5b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/de109d5b

Branch: refs/heads/master
Commit: de109d5b4c7693e935b68233c32e70f3f6b3d513
Parents: 0bdf7fc
Author: Eugene Kirpichov 
Authored: Wed Dec 14 14:29:30 2016 -0800
Committer: Eugene Kirpichov 
Committed: Wed Dec 14 15:34:12 2016 -0800

--
 .../runners/core/ReduceFnContextFactory.java|  16 +--
 .../beam/runners/core/ReduceFnRunnerTest.java   | 133 ++-
 .../beam/sdk/transforms/CombineWithContext.java |   2 +-
 3 files changed, 78 insertions(+), 73 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/de109d5b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
index c5bda9b..c71897d 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
@@ -28,7 +28,6 @@ import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.ActiveWindowSet;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.TimeDomain;
@@ -98,11 +97,7 @@ class ReduceFnContextFactory {
 activeWindows,
 windowingStrategy.getWindowFn().windowCoder(),
 stateInternals,
-stateContextFromComponents(
-options,
-sideInputReader,
-window,
-windowingStrategy.getWindowFn()),
+stateContextFromComponents(options, sideInputReader, window),
 style);
   }
 
@@ -512,8 +507,7 @@ class ReduceFnContextFactory {
   private static  StateContext 
stateContextFromComponents(
   @Nullable final PipelineOptions options,
   final SideInputReader sideInputReader,
-  final W mainInputWindow,
-  final WindowFn windowFn) {
+  final W mainInputWindow) {
 if (options == null) {
   return StateContexts.nullContext();
 } else {
@@ -526,7 +520,11 @@ class ReduceFnContextFactory {
 
 @Override
 public  T sideInput(PCollectionView view) {
-  return sideInputReader.get(view, 
windowFn.getSideInputWindow(mainInputWindow));
+  return sideInputReader.get(
+  view,
+  view.getWindowingStrategyInternal()
+  .getWindowFn()
+  .getSideInputWindow(mainInputWindow));
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/de109d5b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
index ba57567..4abfc9a 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
@@ -17,13 +17,14 @@
  */
 package org.apache.beam.runners.core;
 
-import static com.google.common.base.Preconditions.checkArgument;
 import static 
org.apache.beam.runners.core.WindowMatchers.isSingleWindowedValue;
 import static org.apache.beam.runners.core.WindowMatchers.isWindowedValue;
+import static org.hamcrest.Matchers.anyOf;
 import static org.hamcrest.Matchers.contains;
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.emptyIterable;
 import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertThat;
@@ -36,7 +37,6 @@ import static org.mockito.Mockito.when;
 import static org.mockito.Mockito.withSettings;
 
 import com.google.common.collect.Iterables;
-import java.util.Iterator;
 import java.util.List;
 import org.apache.beam.runners.co

[jira] [Commented] (BEAM-1154) ReduceFnRunner fetches side input from the wrong window

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752305#comment-15752305
 ] 

ASF GitHub Bot commented on BEAM-1154:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1620


> ReduceFnRunner fetches side input from the wrong window
> ---
>
> Key: BEAM-1154
> URL: https://issues.apache.org/jira/browse/BEAM-1154
> Project: Beam
>  Issue Type: Bug
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> This 
> https://github.com/apache/incubator-beam/blame/master/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java#L529
>  is incorrect - I broke it in 
> https://github.com/apache/incubator-beam/commit/90a0d0e13fa0332df805b79b1dc64860d9590217#diff-16edced77586e39a5f31907f4ced51b5R530
> It uses windowing strategy of the main input to do .getSideInputWindow() 
> instead of windowing strategy of the side input.
> The fix is very simple; trying to come up with a test...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1154) ReduceFnRunner fetches side input from the wrong window

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov resolved BEAM-1154.

   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> ReduceFnRunner fetches side input from the wrong window
> ---
>
> Key: BEAM-1154
> URL: https://issues.apache.org/jira/browse/BEAM-1154
> Project: Beam
>  Issue Type: Bug
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
> Fix For: 0.4.0-incubating
>
>
> This 
> https://github.com/apache/incubator-beam/blame/master/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java#L529
>  is incorrect - I broke it in 
> https://github.com/apache/incubator-beam/commit/90a0d0e13fa0332df805b79b1dc64860d9590217#diff-16edced77586e39a5f31907f4ced51b5R530
> It uses windowing strategy of the main input to do .getSideInputWindow() 
> instead of windowing strategy of the side input.
> The fix is very simple; trying to come up with a test...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1630

[kenn]

This closes #1630


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3ca8d2bf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3ca8d2bf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3ca8d2bf

Branch: refs/heads/release-0.4.0-incubating
Commit: 3ca8d2bfc74199ac056a9ab94d6031ef97d746f9
Parents: c09fbd4 d57146f
Author: Kenneth Knowles 
Authored: Thu Dec 15 11:41:56 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 11:41:56 2016 -0800

--
 .../runners/core/ReduceFnContextFactory.java|  16 +--
 .../beam/runners/core/ReduceFnRunnerTest.java   | 133 ++-
 .../beam/sdk/transforms/CombineWithContext.java |   2 +-
 3 files changed, 78 insertions(+), 73 deletions(-)
--

[1/2] incubator-beam git commit: [BEAM-1154] Get side input from proper window in ReduceFn

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating c09fbd43b -> 3ca8d2bfc


[BEAM-1154] Get side input from proper window in ReduceFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d57146f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d57146f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d57146f4

Branch: refs/heads/release-0.4.0-incubating
Commit: d57146f47297af2ad0759216a16898b43e0fa0af
Parents: c09fbd4
Author: Eugene Kirpichov 
Authored: Wed Dec 14 14:29:30 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 11:41:55 2016 -0800

--
 .../runners/core/ReduceFnContextFactory.java|  16 +--
 .../beam/runners/core/ReduceFnRunnerTest.java   | 133 ++-
 .../beam/sdk/transforms/CombineWithContext.java |   2 +-
 3 files changed, 78 insertions(+), 73 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d57146f4/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
index c5bda9b..c71897d 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnContextFactory.java
@@ -28,7 +28,6 @@ import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.ActiveWindowSet;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.TimeDomain;
@@ -98,11 +97,7 @@ class ReduceFnContextFactory {
 activeWindows,
 windowingStrategy.getWindowFn().windowCoder(),
 stateInternals,
-stateContextFromComponents(
-options,
-sideInputReader,
-window,
-windowingStrategy.getWindowFn()),
+stateContextFromComponents(options, sideInputReader, window),
 style);
   }
 
@@ -512,8 +507,7 @@ class ReduceFnContextFactory {
   private static  StateContext 
stateContextFromComponents(
   @Nullable final PipelineOptions options,
   final SideInputReader sideInputReader,
-  final W mainInputWindow,
-  final WindowFn windowFn) {
+  final W mainInputWindow) {
 if (options == null) {
   return StateContexts.nullContext();
 } else {
@@ -526,7 +520,11 @@ class ReduceFnContextFactory {
 
 @Override
 public  T sideInput(PCollectionView view) {
-  return sideInputReader.get(view, 
windowFn.getSideInputWindow(mainInputWindow));
+  return sideInputReader.get(
+  view,
+  view.getWindowingStrategyInternal()
+  .getWindowFn()
+  .getSideInputWindow(mainInputWindow));
 }
 
 @Override

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d57146f4/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
index ba57567..4abfc9a 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
@@ -17,13 +17,14 @@
  */
 package org.apache.beam.runners.core;
 
-import static com.google.common.base.Preconditions.checkArgument;
 import static 
org.apache.beam.runners.core.WindowMatchers.isSingleWindowedValue;
 import static org.apache.beam.runners.core.WindowMatchers.isWindowedValue;
+import static org.hamcrest.Matchers.anyOf;
 import static org.hamcrest.Matchers.contains;
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.emptyIterable;
 import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.greaterThanOrEqualTo;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertThat;
@@ -36,7 +37,6 @@ import static org.mockito.Mockito.when;
 import static org.mockito.Mockito.withSettings;
 
 import com.google.common.collect.Iterables;
-import java.util.Iterator;
 import java.util.List;

[2/2] incubator-beam git commit: Closes #1400

[dhalperi]

Closes #1400


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3e1a6281
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3e1a6281
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3e1a6281

Branch: refs/heads/master
Commit: 3e1a62815ca467951647788d59c00921bd02803a
Parents: 3c4b693 5190083
Author: Dan Halperin 
Authored: Thu Dec 15 11:46:03 2016 -0800
Committer: Dan Halperin 
Committed: Thu Dec 15 11:46:03 2016 -0800

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|  48 +-
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |   7 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 121 -
 .../io/gcp/bigquery/BigQueryTableInserter.java  | 217 -
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 456 +++
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 139 +-
 .../gcp/bigquery/BigQueryTableInserterTest.java | 245 --
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   |  50 +-
 8 files changed, 655 insertions(+), 628 deletions(-)
--

[GitHub] incubator-beam pull request #1400: [BEAM-1022] Add test coverage for BigQuer...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1400


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-1022] Add testing coverage for BigQuery streaming writes

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3c4b6930e -> 3e1a62815


[BEAM-1022] Add testing coverage for BigQuery streaming writes


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/51900830
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/51900830
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/51900830

Branch: refs/heads/master
Commit: 519008303f9cefd3f8f4a8a7a98a9a79717f57ff
Parents: 3c4b693
Author: Reuven Lax 
Authored: Thu Nov 17 10:57:41 2016 -0800
Committer: Dan Halperin 
Committed: Thu Dec 15 11:45:45 2016 -0800

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java|  48 +-
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |   7 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 121 -
 .../io/gcp/bigquery/BigQueryTableInserter.java  | 217 -
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 456 +++
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 139 +-
 .../gcp/bigquery/BigQueryTableInserterTest.java | 245 --
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   |  50 +-
 8 files changed, 655 insertions(+), 628 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/51900830/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 0be8567..28049ed 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -24,7 +24,6 @@ import static com.google.common.base.Preconditions.checkState;
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;
 import com.google.api.client.json.JsonFactory;
-import com.google.api.services.bigquery.Bigquery;
 import com.google.api.services.bigquery.model.Job;
 import com.google.api.services.bigquery.model.JobConfigurationExtract;
 import com.google.api.services.bigquery.model.JobConfigurationLoad;
@@ -33,6 +32,7 @@ import 
com.google.api.services.bigquery.model.JobConfigurationTableCopy;
 import com.google.api.services.bigquery.model.JobReference;
 import com.google.api.services.bigquery.model.JobStatistics;
 import com.google.api.services.bigquery.model.JobStatus;
+import com.google.api.services.bigquery.model.Table;
 import com.google.api.services.bigquery.model.TableReference;
 import com.google.api.services.bigquery.model.TableRow;
 import com.google.api.services.bigquery.model.TableSchema;
@@ -1796,8 +1796,8 @@ public class BigQueryIO {
* Does not modify this object.
*/
   public Bound withCreateDisposition(CreateDisposition createDisposition) {
-return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema, 
createDisposition,
-writeDisposition, validate, bigQueryServices);
+return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema,
+createDisposition, writeDisposition, validate, bigQueryServices);
   }
 
   /**
@@ -1806,8 +1806,8 @@ public class BigQueryIO {
* Does not modify this object.
*/
   public Bound withWriteDisposition(WriteDisposition writeDisposition) {
-return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema, 
createDisposition,
-writeDisposition, validate, bigQueryServices);
+return new Bound(name, jsonTableRef, tableRefFunction, jsonSchema,
+createDisposition, writeDisposition, validate, bigQueryServices);
   }
 
   /**
@@ -2136,7 +2136,8 @@ public class BigQueryIO {
   /** Returns the table reference, or {@code null}. */
   @Nullable
   public ValueProvider getTable() {
-return NestedValueProvider.of(jsonTableRef, new 
JsonTableRefToTableRef());
+return jsonTableRef == null ? null :
+NestedValueProvider.of(jsonTableRef, new JsonTableRefToTableRef());
   }
 
   /** Returns {@code true} if table validation is enabled. */
@@ -2550,6 +2551,13 @@ public class BigQueryIO {
 }
   }
 
+  /**
+   * Clear the cached map of created tables. Used for testing.
+   */
+  @VisibleForTesting
+  static void clearCreatedTables() {
+StreamingWriteFn.clearCreatedTables();
+  }
   /
 
   /**
@@ -2585,6 +2593,15 @@ public class BigQueryIO {
   this.bqServices = checkNotNull(bqServices, "bqServices");
 

[jira] [Commented] (BEAM-1022) TableNamespace should not use Java object equality when comparing windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752318#comment-15752318
 ] 

ASF GitHub Bot commented on BEAM-1022:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1400


> TableNamespace should not use Java object equality when comparing windows
> -
>
> Key: BEAM-1022
> URL: https://issues.apache.org/jira/browse/BEAM-1022
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.3.0-incubating
>Reporter: Reuven Lax
>Assignee: Thomas Groh
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1630: Cherry pick #1620 to release-0.4.0-incuba...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1630


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Apex #4

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-1023) Add test coverage for BigQueryIO.Write in streaming mode

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1023.
---
   Resolution: Fixed
Fix Version/s: 0.5.0-incubating

> Add test coverage for BigQueryIO.Write in streaming mode
> 
>
> Key: BEAM-1023
> URL: https://issues.apache.org/jira/browse/BEAM-1023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 0.5.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1023) Add test coverage for BigQueryIO.Write in streaming mode

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752424#comment-15752424
 ] 

Daniel Halperin commented on BEAM-1023:
---

This was implemented by PR #1400, which was accidentally tagged with BEAM-1022 
instead of this issue.

> Add test coverage for BigQueryIO.Write in streaming mode
> 
>
> Key: BEAM-1023
> URL: https://issues.apache.org/jira/browse/BEAM-1023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 0.5.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1023) Add test coverage for BigQueryIO.Write in streaming mode

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752428#comment-15752428
 ] 

Daniel Halperin commented on BEAM-1023:
---

https://github.com/apache/incubator-beam/pull/1400

> Add test coverage for BigQueryIO.Write in streaming mode
> 
>
> Key: BEAM-1023
> URL: https://issues.apache.org/jira/browse/BEAM-1023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 0.5.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #111: [BEAM-1163] Add signature keys to the...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/incubator-beam-site/pull/111

[BEAM-1163] Add signature keys to the vote template (release guide)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/incubator-beam-site BEAM-1163

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/111.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #111


commit 4461b271569ae900372a7b137b66b64480968684
Author: Ismaël Mejía 
Date:   2016-12-15T20:32:09Z

[BEAM-1163] Add signature keys to the vote template (release guide)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1163) Add signature keys to the release guide vote template

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752444#comment-15752444
 ] 

ASF GitHub Bot commented on BEAM-1163:
--

GitHub user iemejia opened a pull request:

https://github.com/apache/incubator-beam-site/pull/111

[BEAM-1163] Add signature keys to the vote template (release guide)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/incubator-beam-site BEAM-1163

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/111.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #111


commit 4461b271569ae900372a7b137b66b64480968684
Author: Ismaël Mejía 
Date:   2016-12-15T20:32:09Z

[BEAM-1163] Add signature keys to the vote template (release guide)




> Add signature keys to the release guide vote template
> -
>
> Key: BEAM-1163
> URL: https://issues.apache.org/jira/browse/BEAM-1163
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Trivial
>
> A small improvement, the idea is to add just the fingerprint of the person 
> who signed the release in the template (for validation purposes:
> The release artifacts are signed with the key with fingerprint XXX
> https://dist.apache.org/repos/dist/release/incubator/beam/KEYS



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Apex #5

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1165) Unexpected file created when checking dependencies on clean repo

[JIRA]

Ismaël Mejía created BEAM-1165:
--

 Summary: Unexpected file created when checking dependencies on 
clean repo
 Key: BEAM-1165
 URL: https://issues.apache.org/jira/browse/BEAM-1165
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 0.5.0-incubating
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía
Priority: Minor


I just found a weird behavior when I was checking for the latest release,
nothing breaking, but when I start with a clean repo clone and I do:

mvn dependency:tree

It creates a new file runners/flink/examples/wordcounts.txt with the
dependencies.

This error happens because maven-dependency-plugin asumes the property output
used by the flink tests as the export file for the command.

Ref.
https://maven.apache.org/plugins/maven-dependency-plugin/tree-mojo.html#output




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1632: [BEAM-1165] Fix unexpected file creation ...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/incubator-beam/pull/1632

[BEAM-1165] Fix unexpected file creation when checking dependencies

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/incubator-beam BEAM-1165

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1632.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1632


commit c283c8c96c9389e5b432307f69adbf3862e8570f
Author: Ismaël Mejía 
Date:   2016-12-15T21:06:18Z

[BEAM-1165] Fix unexpected file creation when checking dependencies




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1165) Unexpected file created when checking dependencies on clean repo

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752520#comment-15752520
 ] 

ASF GitHub Bot commented on BEAM-1165:
--

GitHub user iemejia opened a pull request:

https://github.com/apache/incubator-beam/pull/1632

[BEAM-1165] Fix unexpected file creation when checking dependencies

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/incubator-beam BEAM-1165

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1632.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1632


commit c283c8c96c9389e5b432307f69adbf3862e8570f
Author: Ismaël Mejía 
Date:   2016-12-15T21:06:18Z

[BEAM-1165] Fix unexpected file creation when checking dependencies




> Unexpected file created when checking dependencies on clean repo
> 
>
> Key: BEAM-1165
> URL: https://issues.apache.org/jira/browse/BEAM-1165
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.5.0-incubating
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>
> I just found a weird behavior when I was checking for the latest release,
> nothing breaking, but when I start with a clean repo clone and I do:
> mvn dependency:tree
> It creates a new file runners/flink/examples/wordcounts.txt with the
> dependencies.
> This error happens because maven-dependency-plugin asumes the property output
> used by the flink tests as the export file for the command.
> Ref.
> https://maven.apache.org/plugins/maven-dependency-plugin/tree-mojo.html#output



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-beam git commit: [maven-release-plugin] prepare release v0.4.0-incubating-RC2

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating 3ca8d2bfc -> a29affd05


[maven-release-plugin] prepare release v0.4.0-incubating-RC2


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a29affd0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a29affd0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a29affd0

Branch: refs/heads/release-0.4.0-incubating
Commit: a29affd05cfa3f387fd12296250d61a4bbe78946
Parents: 3ca8d2b
Author: Davor Bonaci 
Authored: Thu Dec 15 13:16:49 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Dec 15 13:16:49 2016 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 33 files changed, 34 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a29affd0/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index e541edb..7333c10 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-incubating-SNAPSHOT
+0.4.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a29affd0/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 8591955..59d8f4d 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-incubating-SNAPSHOT
+0.4.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a29affd0/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index eb42861..65c6f1b 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.4.0-incubating-SNAPSHOT
+0.4.0-incubating
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a29affd0/pom.xml
--
diff --git a/pom.xml b/pom.xml
index e860c7a..0d3db9d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.incubator.apache.org
   2016
 
-  0.4.0-incubating-SNAPSHOT
+  0.4.0-incubating
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary
-release-0.4.0-incubating
+v0.4.0-incubating-RC2
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a29affd0/runners/apex/pom.xml
--
diff --git a/runners/apex

incubator-beam git commit: [maven-release-plugin] rollback changes from release preparation of v0.4.0-incubating-RC2

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/release-0.4.0-incubating a29affd05 -> f753422a3


[maven-release-plugin] rollback changes from release preparation of 
v0.4.0-incubating-RC2


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f753422a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f753422a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f753422a

Branch: refs/heads/release-0.4.0-incubating
Commit: f753422a36ec6cbcf56dd6f69e1c7ba29e53a350
Parents: a29affd
Author: Davor Bonaci 
Authored: Thu Dec 15 13:22:04 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Dec 15 13:22:04 2016 -0800

--
 examples/java/pom.xml | 2 +-
 examples/java8/pom.xml| 2 +-
 examples/pom.xml  | 2 +-
 pom.xml   | 4 ++--
 runners/apex/pom.xml  | 2 +-
 runners/core-java/pom.xml | 2 +-
 runners/direct-java/pom.xml   | 2 +-
 runners/flink/examples/pom.xml| 2 +-
 runners/flink/pom.xml | 2 +-
 runners/flink/runner/pom.xml  | 2 +-
 runners/google-cloud-dataflow-java/pom.xml| 2 +-
 runners/pom.xml   | 2 +-
 runners/spark/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml | 2 +-
 sdks/java/core/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml | 2 +-
 sdks/java/extensions/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml| 2 +-
 sdks/java/io/hdfs/pom.xml | 2 +-
 sdks/java/io/jdbc/pom.xml | 2 +-
 sdks/java/io/jms/pom.xml  | 2 +-
 sdks/java/io/kafka/pom.xml| 2 +-
 sdks/java/io/kinesis/pom.xml  | 2 +-
 sdks/java/io/mongodb/pom.xml  | 2 +-
 sdks/java/io/pom.xml  | 2 +-
 sdks/java/java8tests/pom.xml  | 2 +-
 sdks/java/maven-archetypes/examples-java8/pom.xml | 2 +-
 sdks/java/maven-archetypes/examples/pom.xml   | 2 +-
 sdks/java/maven-archetypes/pom.xml| 2 +-
 sdks/java/maven-archetypes/starter/pom.xml| 2 +-
 sdks/java/pom.xml | 2 +-
 sdks/pom.xml  | 2 +-
 33 files changed, 34 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 7333c10..e541edb 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-incubating
+0.4.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 59d8f4d..8591955 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-examples-parent
-0.4.0-incubating
+0.4.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 65c6f1b..eb42861 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.beam
 beam-parent
-0.4.0-incubating
+0.4.0-incubating-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 0d3db9d..e860c7a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -34,7 +34,7 @@
   http://beam.incubator.apache.org
   2016
 
-  0.4.0-incubating
+  0.4.0-incubating-SNAPSHOT
 
   
 
@@ -48,7 +48,7 @@
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
scm:git:https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
https://git-wip-us.apache.org/repos/asf?p=incubator-beam.git;a=summary
-v0.4.0-incubating-RC2
+release-0.4.0-incubating
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f753422a/runners/apex/pom.xml
--

[incubator-beam] Git Push Summary

[davor]

Repository: incubator-beam
Updated Tags:  refs/tags/v0.4.0-incubating-RC2 [created] 76246ac4f

[jira] [Created] (BEAM-1166) Source.getDefaultOutputCoder() documentation should mention CannotProvideCoderException

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-1166:
--

 Summary: Source.getDefaultOutputCoder() documentation should 
mention CannotProvideCoderException
 Key: BEAM-1166
 URL: https://issues.apache.org/jira/browse/BEAM-1166
 Project: Beam
  Issue Type: Bug
Reporter: Stephen Sisk
Assignee: Stephen Sisk


Knowing that you can throw CannotProviderCoderException is an important part of 
implementing getDefaultOutputCoder

The documentation for PTransform's getDefaultOutputCoder mentions this class, 
and we should do this for the Source class as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1633: [BEAM-450] Shade modules to separate path...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1633

[BEAM-450] Shade modules to separate paths

R: @lukecwik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam shading-package-names

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1633.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1633


commit 0bed1b3c494ec23f3ca18389d7b3bce15e0bd363
Author: Dan Halperin 
Date:   2016-12-15T21:50:39Z

[BEAM-450] Shade modules to separate paths




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-450) Modules are shaded to the same path

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752612#comment-15752612
 ] 

ASF GitHub Bot commented on BEAM-450:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1633

[BEAM-450] Shade modules to separate paths

R: @lukecwik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam shading-package-names

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1633.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1633


commit 0bed1b3c494ec23f3ca18389d7b3bce15e0bd363
Author: Dan Halperin 
Date:   2016-12-15T21:50:39Z

[BEAM-450] Shade modules to separate paths




> Modules are shaded to the same path
> ---
>
> Key: BEAM-450
> URL: https://issues.apache.org/jira/browse/BEAM-450
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>  Labels: newbie, starter
>
> Right now multiple modules are using the same repackaged path. We should be 
> using per-artifact paths so that they don't conflict.
> One proposal was simply to adopt 
> {{${project.groupId}.${project.artifactId}.repackaged}} as the shading 
> location. If it works.
> This is a good starter issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[10/10] incubator-beam git commit: This closes #1565

[kenn]

This closes #1565

  Renames ParDo.getNewFn to getFn
  Moves DoFnAdapters to runners-core
  Removes unused code from NoOpOldDoFn
  Removes ArgumentProvider.windowingInternals
  Removes code for wrapping DoFn as an OldDoFn
  Removes OldDoFn from ParDo
  Pushes uses of OldDoFn deeper inside Flink runner
  Remove ParDo.of(OldDoFn) from Apex runner
  Converts all easy OldDoFns to DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5a3ace4a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5a3ace4a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5a3ace4a

Branch: refs/heads/master
Commit: 5a3ace4a7c786938e5286e921f3afcd23b26de26
Parents: 3e1a628 6b502fc
Author: Kenneth Knowles 
Authored: Thu Dec 15 14:04:03 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 14:04:03 2016 -0800

--
 .../apache/beam/runners/apex/ApexRunner.java|  57 +--
 .../translation/ApexPipelineTranslator.java |   2 +
 .../translation/ParDoBoundMultiTranslator.java  |   6 +-
 .../apex/translation/ParDoBoundTranslator.java  |   6 +-
 .../apex/translation/WindowBoundTranslator.java |  78 +++
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../operators/ApexParDoOperator.java|  25 +-
 .../FlattenPCollectionTranslatorTest.java   |  15 +-
 .../translation/GroupByKeyTranslatorTest.java   |  21 +-
 .../translation/ParDoBoundTranslatorTest.java   |  38 +-
 .../translation/ReadUnboundTranslatorTest.java  |  15 +-
 .../apache/beam/runners/core/AssignWindows.java |  46 --
 .../apache/beam/runners/core/DoFnAdapters.java  | 344 +
 .../beam/runners/core/SimpleDoFnRunner.java |  57 ---
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   4 +-
 .../beam/runners/core/SplittableParDo.java  |  11 +-
 .../core/GroupAlsoByWindowsProperties.java  |   2 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |   2 +-
 .../direct/ParDoMultiOverrideFactory.java   |   2 +-
 .../ParDoSingleViaMultiOverrideFactory.java |   4 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   4 +-
 .../apache/beam/runners/flink/FlinkRunner.java  |  10 +-
 .../FlinkBatchTransformTranslators.java |  13 +-
 .../FlinkStreamingTransformTranslators.java |   4 +-
 .../functions/FlinkDoFnFunction.java|  10 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |  10 +-
 .../functions/FlinkProcessContextBase.java  |   2 +-
 .../wrappers/streaming/DoFnOperator.java|  43 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |   8 +-
 .../beam/runners/flink/PipelineOptionsTest.java |   6 +-
 .../flink/streaming/DoFnOperatorTest.java   |  13 +-
 .../flink/streaming/GroupByNullKeyTest.java |  18 +-
 .../streaming/TopWikipediaSessionsITCase.java   |  10 +-
 .../dataflow/DataflowPipelineTranslator.java|   8 +-
 .../spark/translation/SparkAssignWindowFn.java  |   2 +-
 .../spark/translation/TransformTranslator.java  |   4 +-
 .../streaming/StreamingTransformTranslator.java |   4 +-
 .../beam/sdk/AggregatorPipelineExtractor.java   |   4 +-
 .../sdk/transforms/AggregatorRetriever.java |  13 +-
 .../beam/sdk/transforms/DoFnAdapters.java   | 504 ---
 .../apache/beam/sdk/transforms/DoFnTester.java  |   7 -
 .../org/apache/beam/sdk/transforms/OldDoFn.java | 297 +--
 .../org/apache/beam/sdk/transforms/ParDo.java   | 173 +--
 .../sdk/transforms/reflect/DoFnInvoker.java |  20 -
 .../sdk/transforms/reflect/DoFnInvokers.java| 141 +-
 .../sdk/AggregatorPipelineExtractorTest.java|  12 +-
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java |  74 +--
 .../apache/beam/sdk/transforms/OldDoFnTest.java | 125 ++---
 .../transforms/reflect/DoFnInvokersTest.java|  42 --
 49 files changed, 689 insertions(+), 1629 deletions(-)
--

[03/10] incubator-beam git commit: Remove ParDo.of(OldDoFn) from Apex runner

[kenn]

Remove ParDo.of(OldDoFn) from Apex runner

The only such usage was of AssignWindowsDoFn. Now, instead, it is
instantiated directly using a new translator for Window.Bound.

This change also separates the overloads of ApexParDoOperator for old
and new DoFn, to make the OldDoFn overload easier to track and later
remove.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/af616d97
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/af616d97
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/af616d97

Branch: refs/heads/master
Commit: af616d9741b19d0a7705df6fe075be1509aa659f
Parents: f5f329e
Author: Eugene Kirpichov 
Authored: Fri Dec 9 16:31:42 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:55:24 2016 -0800

--
 .../apache/beam/runners/apex/ApexRunner.java| 46 +---
 .../translation/ApexPipelineTranslator.java |  2 +
 .../translation/ParDoBoundMultiTranslator.java  |  4 +-
 .../apex/translation/ParDoBoundTranslator.java  |  4 +-
 .../apex/translation/WindowBoundTranslator.java | 78 
 .../operators/ApexParDoOperator.java| 25 ++-
 .../translation/ParDoBoundTranslatorTest.java   |  3 +-
 .../apache/beam/runners/core/AssignWindows.java | 46 
 .../spark/translation/SparkAssignWindowFn.java  |  2 +-
 9 files changed, 108 insertions(+), 102 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/af616d97/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
index e5bde46..f12ebef 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
@@ -32,7 +32,6 @@ import org.apache.apex.api.Launcher;
 import org.apache.apex.api.Launcher.AppHandle;
 import org.apache.apex.api.Launcher.LaunchMode;
 import org.apache.beam.runners.apex.translation.ApexPipelineTranslator;
-import org.apache.beam.runners.core.AssignWindows;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderRegistry;
@@ -46,9 +45,6 @@ import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.Window;
-import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.PCollectionViews;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.PCollection;
@@ -90,10 +86,7 @@ public class ApexRunner extends 
PipelineRunner {
   public  OutputT apply(
   PTransform transform, InputT input) {
 
-if (Window.Bound.class.equals(transform.getClass())) {
-  return (OutputT) ((PCollection) input).apply(
-  new AssignWindowsAndSetStrategy((Window.Bound) transform));
-} else if (Create.Values.class.equals(transform.getClass())) {
+if (Create.Values.class.equals(transform.getClass())) {
   return (OutputT) PCollection
   .createPrimitiveOutputInternal(
   input.getPipeline(),
@@ -162,43 +155,6 @@ public class ApexRunner extends 
PipelineRunner {
 
   }
 
-  /**
-   * copied from DirectPipelineRunner.
-   * used to replace Window.Bound till equivalent function is added in Apex
-   */
-  private static class AssignWindowsAndSetStrategy
-  extends PTransform, PCollection> {
-
-private final Window.Bound wrapped;
-
-public AssignWindowsAndSetStrategy(Window.Bound wrapped) {
-  this.wrapped = wrapped;
-}
-
-@Override
-public PCollection expand(PCollection input) {
-  WindowingStrategy outputStrategy =
-  wrapped.getOutputStrategyInternal(input.getWindowingStrategy());
-
-  WindowFn windowFn =
-  (WindowFn) outputStrategy.getWindowFn();
-
-  // If the Window.Bound transform only changed parts other than the 
WindowFn, then
-  // we skip AssignWindows even though it should be harmless in a perfect 
world.
-  // The world is not perfect, and a GBK may have set it to InvalidWindows 
to forcibly
-  // crash if another GBK is performed without explicitly setting the 
WindowFn. So we skip
-  // AssignWindows in this case.
-  if (wrapped.getWindowFn() == null) {
-return input.apply("Identity", ParDo.of(new IdentityFn()))
-.setWindowingStrategyInternal(outputStrategy);
-  } else {
-retur

[05/10] incubator-beam git commit: Removes OldDoFn from ParDo

[kenn]

Removes OldDoFn from ParDo


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e9e53c5d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e9e53c5d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e9e53c5d

Branch: refs/heads/master
Commit: e9e53c5d037561aa4dcacfcde69d76a03f3a1571
Parents: 8330bfa
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:13:43 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:58:43 2016 -0800

--
 .../org/apache/beam/sdk/transforms/ParDo.java   | 167 +++
 .../apache/beam/sdk/transforms/OldDoFnTest.java | 125 --
 2 files changed, 55 insertions(+), 237 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e9e53c5d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index 167f5fa..d2149c0 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -27,7 +27,6 @@ import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.CannotProvideCoderException;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.runners.PipelineRunner;
-import org.apache.beam.sdk.transforms.OldDoFn.RequiresWindowAccess;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.DisplayData.Builder;
 import org.apache.beam.sdk.transforms.display.HasDisplayData;
@@ -530,23 +529,6 @@ public class ParDo {
 return new Unbound().of(fn, displayDataForFn(fn));
   }
 
-  /**
-   * Creates a {@link ParDo} {@link PTransform} that will invoke the
-   * given {@link OldDoFn} function.
-   *
-   * The resulting {@link PTransform PTransform's} types have been bound, 
with the
-   * input being a {@code PCollection} and the output a
-   * {@code PCollection}, inferred from the types of the argument
-   * {@code OldDoFn}. It is ready to be applied, or further
-   * properties can be set on it first.
-   *
-   * @deprecated please port your {@link OldDoFn} to a {@link DoFn}
-   */
-  @Deprecated
-  public static  Bound of(OldDoFn fn) {
-return new Unbound().of(fn, displayDataForFn(fn));
-  }
-
   private static  DisplayData.ItemSpec> 
displayDataForFn(T fn) {
 return DisplayData.item("fn", fn.getClass()).withLabel("Transform 
Function");
   }
@@ -557,12 +539,7 @@ public class ParDo {
* the {@link PCollection}.
*/
   private static  void validateWindowType(
-  PCollection input, Serializable fn) {
-// No validation for OldDoFn
-if (!(fn instanceof DoFn)) {
-  return;
-}
-
+  PCollection input, DoFn fn) {
 DoFnSignature signature = DoFnSignatures.getSignature((Class) 
fn.getClass());
 
 TypeDescriptor actualWindowT =
@@ -609,10 +586,6 @@ public class ParDo {
 }
   }
 
-  private static  OldDoFn adapt(DoFn fn) {
-return DoFnAdapters.toOldDoFn(fn);
-  }
-
   /**
* An incomplete {@link ParDo} transform, with unbound input/output types.
*
@@ -688,24 +661,9 @@ public class ParDo {
   return new UnboundMulti<>(name, sideInputs, mainOutputTag, 
sideOutputTags);
 }
 
-/**
- * Returns a new {@link ParDo} {@link PTransform} that's like this
- * transform but that will invoke the given {@link OldDoFn}
- * function, and that has its input and output types bound. Does
- * not modify this transform. The resulting {@link PTransform} is
- * sufficiently specified to be applied, but more properties can
- * still be specified.
- *
- * @deprecated please port your {@link OldDoFn} to a {@link DoFn}
- */
-@Deprecated
-public  Bound of(OldDoFn oldFn) {
-  return of(oldFn, displayDataForFn(oldFn));
-}
-
 private  Bound of(
-Serializable originalFn, DisplayData.ItemSpec> 
fnDisplayData) {
-  return new Bound<>(name, originalFn, sideInputs, fnDisplayData);
+DoFn doFn, DisplayData.ItemSpec> 
fnDisplayData) {
+  return new Bound<>(name, doFn, sideInputs, fnDisplayData);
 }
   }
 
@@ -725,12 +683,12 @@ public class ParDo {
   extends PTransform, PCollection> {
 // Inherits name.
 private final List> sideInputs;
-private final Serializable fn;
+private final DoFn fn;
 private final DisplayData.ItemSpec> fnDisplayData;
 
 Bound(
 String name,
-Serializable fn,
+DoFn fn,
 List> sideInputs,
 DisplayData.ItemSpec> fnDisplayData) {
   super(name);
@@ -787,7 +745,7 @@ public class ParDo {
 @Overr

[09/10] incubator-beam git commit: Renames ParDo.getNewFn to getFn

[kenn]

Renames ParDo.getNewFn to getFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6b502fc1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6b502fc1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6b502fc1

Branch: refs/heads/master
Commit: 6b502fc111af266c7b1a0e6f7d473c36f57281a2
Parents: 33ed323
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:29:41 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:59:11 2016 -0800

--
 .../translation/ParDoBoundMultiTranslator.java  |  2 +-
 .../apex/translation/ParDoBoundTranslator.java  |  2 +-
 .../beam/runners/core/SplittableParDo.java  |  4 ++--
 .../runners/direct/ParDoEvaluatorFactory.java   |  2 +-
 .../direct/ParDoMultiOverrideFactory.java   |  2 +-
 .../ParDoSingleViaMultiOverrideFactory.java |  4 ++--
 .../direct/StatefulParDoEvaluatorFactory.java   |  4 ++--
 .../FlinkBatchTransformTranslators.java |  4 ++--
 .../FlinkStreamingTransformTranslators.java | 12 +--
 .../dataflow/DataflowPipelineTranslator.java|  8 +++
 .../spark/translation/TransformTranslator.java  |  4 ++--
 .../streaming/StreamingTransformTranslator.java |  4 ++--
 .../beam/sdk/AggregatorPipelineExtractor.java   |  4 ++--
 .../org/apache/beam/sdk/transforms/ParDo.java   | 22 ++--
 .../sdk/AggregatorPipelineExtractorTest.java| 12 +--
 15 files changed, 45 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6b502fc1/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
index 574ce8f..bff7652 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundMultiTranslator.java
@@ -53,7 +53,7 @@ class ParDoBoundMultiTranslator
 
   @Override
   public void translate(ParDo.BoundMulti transform, 
TranslationContext context) {
-DoFn doFn = transform.getNewFn();
+DoFn doFn = transform.getFn();
 DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
 
 if (signature.stateDeclarations().size() > 0) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6b502fc1/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
index de78628..3b6eb6e 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/ParDoBoundTranslator.java
@@ -40,7 +40,7 @@ class ParDoBoundTranslator
 
   @Override
   public void translate(ParDo.Bound transform, 
TranslationContext context) {
-DoFn doFn = transform.getNewFn();
+DoFn doFn = transform.getFn();
 DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
 
 if (signature.stateDeclarations().size() > 0) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6b502fc1/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
index 720db63..f8d12ec 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
@@ -104,7 +104,7 @@ public class SplittableParDo
 checkNotNull(parDo, "parDo must not be null");
 this.parDo = parDo;
 checkArgument(
-
DoFnSignatures.getSignature(parDo.getNewFn().getClass()).processElement().isSplittable(),
+
DoFnSignatures.getSignature(parDo.getFn().getClass()).processElement().isSplittable(),
 "fn must be a splittable DoFn");
   }
 
@@ -114,7 +114,7 @@ public class SplittableParDo
   }
 
   private PCollectionTuple applyTyped(PCollection input) {
-DoFn fn = parDo.getNewFn();
+DoFn fn = parDo.getFn();
 Coder restrictionCoder =
 DoFnInvokers.invokerFor(fn)
 

[07/10] incubator-beam git commit: Removes ArgumentProvider.windowingInternals

[kenn]

Removes ArgumentProvider.windowingInternals


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f3e8a038
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f3e8a038
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f3e8a038

Branch: refs/heads/master
Commit: f3e8a0383bf9cb3f9452e0364f7deba113cadff9
Parents: a22de15
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:23:15 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:58:43 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java | 57 
 .../beam/runners/core/SplittableParDo.java  |  7 ---
 .../beam/sdk/transforms/DoFnAdapters.java   | 14 -
 .../apache/beam/sdk/transforms/DoFnTester.java  |  7 ---
 .../sdk/transforms/reflect/DoFnInvoker.java | 20 ---
 .../transforms/reflect/DoFnInvokersTest.java|  6 ---
 6 files changed, 111 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3e8a038/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index af7f5ca..041cdde 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -52,13 +52,10 @@ import 
org.apache.beam.sdk.util.ExecutionContext.StepContext;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.SystemDoFnInternal;
 import org.apache.beam.sdk.util.Timer;
-import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.State;
-import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateNamespace;
 import org.apache.beam.sdk.util.state.StateNamespaces;
 import org.apache.beam.sdk.util.state.StateSpec;
@@ -420,11 +417,6 @@ public class SimpleDoFnRunner implements 
DoFnRunner windowingInternals() {
-  throw new UnsupportedOperationException("WindowingInternals are 
unsupported.");
-}
-
-@Override
 public  RestrictionTracker 
restrictionTracker() {
   throw new UnsupportedOperationException(
   "Cannot access RestrictionTracker outside of @ProcessElement 
method.");
@@ -634,54 +626,5 @@ public class SimpleDoFnRunner implements 
DoFnRunner windowingInternals() {
-  return new WindowingInternals() {
-@Override
-public Collection windows() {
-  return windowedValue.getWindows();
-}
-
-@Override
-public PaneInfo pane() {
-  return windowedValue.getPane();
-}
-
-@Override
-public TimerInternals timerInternals() {
-  return context.stepContext.timerInternals();
-}
-
-@Override
-public StateInternals stateInternals() {
-  return stepContext.stateInternals();
-}
-
-@Override
-public void outputWindowedValue(
-OutputT output,
-Instant timestamp,
-Collection windows,
-PaneInfo pane) {
-  throw new UnsupportedOperationException("A DoFn cannot output to a 
different window");
-}
-
-@Override
-public  void sideOutputWindowedValue(
-TupleTag tag,
-SideOutputT output,
-Instant timestamp,
-Collection windows,
-PaneInfo pane) {
-  throw new UnsupportedOperationException(
-  "A DoFn cannot side output to a different window");
-}
-
-@Override
-public  T sideInput(PCollectionView view, BoundedWindow 
sideInputWindow) {
-  return context.sideInput(view, sideInputWindow);
-}
-  };
-}
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f3e8a038/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
index 8a9bfcd..720db63 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
@@ -53,7 +53,6 @@ import org.

[06/10] incubator-beam git commit: Removes unused code from NoOpOldDoFn

[kenn]

Removes unused code from NoOpOldDoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ca1dd7a3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ca1dd7a3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ca1dd7a3

Branch: refs/heads/master
Commit: ca1dd7a3a4a01e2696a8131809c5798aef55d6a0
Parents: f3e8a03
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:24:23 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:58:43 2016 -0800

--
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java | 72 
 1 file changed, 72 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ca1dd7a3/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpOldDoFn.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpOldDoFn.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpOldDoFn.java
index 302b66a..504480b 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpOldDoFn.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/NoOpOldDoFn.java
@@ -19,10 +19,6 @@ package org.apache.beam.sdk.transforms;
 
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.WindowingInternals;
-import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
 import org.joda.time.Instant;
 
@@ -46,13 +42,6 @@ class NoOpOldDoFn extends OldDoFn {
   }
 
   /**
-   * Returns a new NoOp Process Context.
-   */
-  public OldDoFn.ProcessContext processContext() {
-return new NoOpDoFnProcessContext();
-  }
-
-  /**
* A {@link OldDoFn.Context} that does nothing and returns exclusively null.
*/
   private class NoOpDoFnContext extends OldDoFn.Context {
@@ -79,65 +68,4 @@ class NoOpOldDoFn extends OldDoFn {
   return null;
 }
   }
-
-  /**
-   * A {@link OldDoFn.ProcessContext} that does nothing and returns exclusively
-   * null.
-   */
-  private class NoOpDoFnProcessContext extends OldDoFn.ProcessContext {
-@Override
-public InputT element() {
-  return null;
-}
-
-@Override
-public  T sideInput(PCollectionView view) {
-  return null;
-}
-
-@Override
-public Instant timestamp() {
-  return null;
-}
-
-@Override
-public BoundedWindow window() {
-  return null;
-}
-
-@Override
-public PaneInfo pane() {
-  return null;
-}
-
-@Override
-public WindowingInternals windowingInternals() {
-  return null;
-}
-
-@Override
-public PipelineOptions getPipelineOptions() {
-  return null;
-}
-
-@Override
-public void output(OutputT output) {}
-
-@Override
-public void outputWithTimestamp(OutputT output, Instant timestamp) {}
-
-@Override
-public  void sideOutput(TupleTag tag, T output) {}
-
-@Override
-public  void sideOutputWithTimestamp(TupleTag tag, T output,
-Instant timestamp) {}
-
-@Override
-protected  Aggregator
-createAggregatorInternal(String name, CombineFn combiner) {
-  return null;
-}
-
-  }
 }

[08/10] incubator-beam git commit: Moves DoFnAdapters to runners-core

[kenn]

Moves DoFnAdapters to runners-core


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/33ed3238
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/33ed3238
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/33ed3238

Branch: refs/heads/master
Commit: 33ed3238e2b3899cff061be3056c5cc29fc60a04
Parents: ca1dd7a
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:28:16 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:59:11 2016 -0800

--
 .../apex/translation/WindowBoundTranslator.java |   2 +-
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../operators/ApexParDoOperator.java|   2 +-
 .../apache/beam/runners/core/DoFnAdapters.java  | 344 +++
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   4 +-
 .../core/GroupAlsoByWindowsProperties.java  |   2 +-
 .../functions/FlinkDoFnFunction.java|   2 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |   2 +-
 .../functions/FlinkProcessContextBase.java  |   2 +-
 .../wrappers/streaming/DoFnOperator.java|   2 +-
 .../sdk/transforms/AggregatorRetriever.java |  13 +-
 .../beam/sdk/transforms/DoFnAdapters.java   | 340 --
 .../org/apache/beam/sdk/transforms/OldDoFn.java |   2 +-
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java |   2 +-
 14 files changed, 367 insertions(+), 354 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33ed3238/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
index 33b9269..ef049e1 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
@@ -22,8 +22,8 @@ import java.util.Collections;
 import org.apache.beam.runners.apex.ApexPipelineOptions;
 import org.apache.beam.runners.apex.translation.operators.ApexParDoOperator;
 import org.apache.beam.runners.core.AssignWindowsDoFn;
+import org.apache.beam.runners.core.DoFnAdapters;
 import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.DoFnAdapters;
 import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.Window;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33ed3238/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 48ac177..4af7ff0 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -413,7 +413,7 @@ public class ApexGroupByKeyOperator implements 
Operator {
 }
 
 @Override
-protected  Aggregator 
createAggregatorInternal(
+public  Aggregator 
createAggregatorInternal(
 String name, Combine.CombineFn combiner) {
   throw new UnsupportedOperationException();
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33ed3238/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
index 08f062d..1e76949 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
@@ -37,6 +37,7 @@ import 
org.apache.beam.runners.apex.translation.utils.ApexStreamTuple;
 import org.apache.beam.runners.apex.translation.utils.NoOpStepContext;
 import 
org.apache.beam.runners.apex.translation.utils.SerializablePipelineOptions;
 import 
org.apache.beam.runners.apex.translation.utils.ValueAndCoderKryoSerializable;
+import org.apache.beam.runners.core.DoFnAdapters;
 import org.apache

[04/10] incubator-beam git commit: Removes code for wrapping DoFn as an OldDoFn

[kenn]

Removes code for wrapping DoFn as an OldDoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a22de150
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a22de150
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a22de150

Branch: refs/heads/master
Commit: a22de15012c51e8b7e31143021f0a298e093bf51
Parents: e9e53c5
Author: Eugene Kirpichov 
Authored: Fri Dec 9 17:21:40 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:58:43 2016 -0800

--
 .../beam/sdk/transforms/DoFnAdapters.java   | 150 --
 .../org/apache/beam/sdk/transforms/OldDoFn.java | 295 +--
 .../sdk/transforms/reflect/DoFnInvokers.java| 141 +
 .../transforms/reflect/DoFnInvokersTest.java|  36 ---
 4 files changed, 11 insertions(+), 611 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a22de150/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
index e15b08b..d1c40a6 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
@@ -18,8 +18,6 @@
 package org.apache.beam.sdk.transforms;
 
 import java.io.IOException;
-import java.util.Collection;
-import javax.annotation.Nullable;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
 import org.apache.beam.sdk.transforms.DoFn.Context;
@@ -38,7 +36,6 @@ import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.state.State;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.TypeDescriptor;
 import org.joda.time.Duration;
 import org.joda.time.Instant;
 
@@ -53,18 +50,6 @@ public class DoFnAdapters {
   /** Should not be instantiated. */
   private DoFnAdapters() {}
 
-  /**
-   * If this is an {@link OldDoFn} produced via {@link #toOldDoFn}, returns 
the class of the
-   * original {@link DoFn}, otherwise returns {@code fn.getClass()}.
-   */
-  public static Class getDoFnClass(OldDoFn fn) {
-if (fn instanceof SimpleDoFnAdapter) {
-  return ((SimpleDoFnAdapter) fn).fn.getClass();
-} else {
-  return fn.getClass();
-}
-  }
-
   /** Creates an {@link OldDoFn} that delegates to the {@link DoFn}. */
   @SuppressWarnings({"unchecked", "rawtypes"})
   public static  OldDoFn 
toOldDoFn(DoFn fn) {
@@ -76,126 +61,6 @@ public class DoFnAdapters {
 }
   }
 
-  /** Creates a {@link OldDoFn.ProcessContext} from a {@link 
DoFn.ProcessContext}. */
-  public static  OldDoFn.ProcessContext 
adaptProcessContext(
-  OldDoFn fn,
-  final DoFn.ProcessContext c,
-  final DoFnInvoker.ArgumentProvider extra) {
-return fn.new ProcessContext() {
-  @Override
-  public InputT element() {
-return c.element();
-  }
-
-  @Override
-  public  T sideInput(PCollectionView view) {
-return c.sideInput(view);
-  }
-
-  @Override
-  public Instant timestamp() {
-return c.timestamp();
-  }
-
-  @Override
-  public BoundedWindow window() {
-return extra.window();
-  }
-
-  @Override
-  public PaneInfo pane() {
-return c.pane();
-  }
-
-  @Override
-  public WindowingInternals windowingInternals() {
-return extra.windowingInternals();
-  }
-
-  @Override
-  public PipelineOptions getPipelineOptions() {
-return c.getPipelineOptions();
-  }
-
-  @Override
-  public void output(OutputT output) {
-c.output(output);
-  }
-
-  @Override
-  public void outputWithTimestamp(OutputT output, Instant timestamp) {
-c.outputWithTimestamp(output, timestamp);
-  }
-
-  @Override
-  public  void sideOutput(TupleTag tag, T output) {
-c.sideOutput(tag, output);
-  }
-
-  @Override
-  public  void sideOutputWithTimestamp(TupleTag tag, T output, 
Instant timestamp) {
-c.sideOutputWithTimestamp(tag, output, timestamp);
-  }
-
-  @Override
-  protected  Aggregator 
createAggregatorInternal(
-  String name, CombineFn combiner) {
-return c.createAggregator(name, combiner);
-  }
-};
-  }
-
-  /** Creates a {@link OldDoFn.ProcessContext} from a {@link 
DoFn.ProcessContext}. */
-  public static  OldDoFn.Context 
adaptContext(
-  OldDoFn fn,
-  final DoFn.Context c) {
-return fn.new Context() {
- 

[01/10] incubator-beam git commit: Converts all easy OldDoFns to DoFn

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3e1a62815 -> 5a3ace4a7


Converts all easy OldDoFns to DoFn


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f5f329ee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f5f329ee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f5f329ee

Branch: refs/heads/master
Commit: f5f329eee4e4a446dafe15b1c42a8f0972360fbc
Parents: 3e1a628
Author: Eugene Kirpichov 
Authored: Fri Dec 9 16:17:46 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:48:27 2016 -0800

--
 .../apache/beam/runners/apex/ApexRunner.java| 11 ++
 .../FlattenPCollectionTranslatorTest.java   | 15 +++-
 .../translation/GroupByKeyTranslatorTest.java   | 21 ---
 .../translation/ParDoBoundTranslatorTest.java   | 39 ++--
 .../translation/ReadUnboundTranslatorTest.java  | 15 +++-
 .../apache/beam/runners/flink/FlinkRunner.java  | 10 ++---
 .../beam/runners/flink/PipelineOptionsTest.java | 11 +++---
 .../flink/streaming/DoFnOperatorTest.java   | 19 +-
 .../flink/streaming/GroupByNullKeyTest.java | 18 -
 .../streaming/TopWikipediaSessionsITCase.java   | 10 ++---
 10 files changed, 75 insertions(+), 94 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f5f329ee/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
index 899efa3..e5bde46 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
@@ -22,13 +22,11 @@ import com.datatorrent.api.Context.DAGContext;
 import com.datatorrent.api.DAG;
 import com.datatorrent.api.StreamingApplication;
 import com.google.common.base.Throwables;
-
 import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
 import java.util.concurrent.atomic.AtomicReference;
-
 import org.apache.apex.api.EmbeddedAppLauncher;
 import org.apache.apex.api.Launcher;
 import org.apache.apex.api.Launcher.AppHandle;
@@ -45,7 +43,6 @@ import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.Combine;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
@@ -245,10 +242,10 @@ public class ApexRunner extends 
PipelineRunner {
 }
   }
 
-  private static class WrapAsList extends OldDoFn> {
-@Override
+  private static class WrapAsList extends DoFn> {
+@ProcessElement
 public void processElement(ProcessContext c) {
-  c.output(Arrays.asList(c.element()));
+  c.output(Collections.singletonList(c.element()));
 }
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f5f329ee/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
--
diff --git 
a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
 
b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
index 6b62a58..f5abc34 100644
--- 
a/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
+++ 
b/runners/apex/src/test/java/org/apache/beam/runners/apex/translation/FlattenPCollectionTranslatorTest.java
@@ -19,12 +19,11 @@
 package org.apache.beam.runners.apex.translation;
 
 import com.google.common.collect.Sets;
-
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
 import java.util.Set;
-
 import org.apache.beam.runners.apex.ApexPipelineOptions;
 import org.apache.beam.runners.apex.ApexRunner;
 import org.apache.beam.runners.apex.ApexRunnerResult;
@@ -32,8 +31,8 @@ import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollect

[02/10] incubator-beam git commit: Pushes uses of OldDoFn deeper inside Flink runner

[kenn]

Pushes uses of OldDoFn deeper inside Flink runner

In particular, various DoFnOperator's now take a regular DoFn
rather than an OldDoFn, and convert it to an OldDoFn internally.

This allows to remove uses of ParDo.getFn() returning OldDoFn.

The only case where the OldDoFn inside a DoFnOperator is actually an
OldDoFn rather than DoFn in disguise is now WindowDoFnOperator, which
overrides getDoFn to return an actual GABW OldDoFn.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8330bfa7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8330bfa7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8330bfa7

Branch: refs/heads/master
Commit: 8330bfa74cd72e51a29649745e87a4f1a6e5ffa1
Parents: af616d9
Author: Eugene Kirpichov 
Authored: Fri Dec 9 16:47:01 2016 -0800
Committer: Eugene Kirpichov 
Committed: Thu Dec 15 13:55:24 2016 -0800

--
 .../FlinkBatchTransformTranslators.java |  9 +---
 .../FlinkStreamingTransformTranslators.java |  8 ++--
 .../functions/FlinkDoFnFunction.java| 10 +++--
 .../functions/FlinkMultiOutputDoFnFunction.java | 10 +++--
 .../wrappers/streaming/DoFnOperator.java| 43 
 .../wrappers/streaming/WindowDoFnOperator.java  |  8 ++--
 .../beam/runners/flink/PipelineOptionsTest.java |  5 +--
 .../flink/streaming/DoFnOperatorTest.java   |  8 ++--
 8 files changed, 63 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8330bfa7/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
index 9ac907f..497b293 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
@@ -50,7 +50,6 @@ import org.apache.beam.sdk.transforms.CombineFnBase;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
@@ -523,8 +522,6 @@ class FlinkBatchTransformTranslators {
   DataSet> inputDataSet =
   context.getInputDataSet(context.getInput(transform));
 
-  final OldDoFn oldDoFn = transform.getFn();
-
   TypeInformation> typeInformation =
   context.getTypeInfo(context.getOutput(transform));
 
@@ -539,7 +536,7 @@ class FlinkBatchTransformTranslators {
 
   FlinkDoFnFunction doFnWrapper =
   new FlinkDoFnFunction<>(
-  oldDoFn,
+  doFn,
   context.getOutput(transform).getWindowingStrategy(),
   sideInputStrategies,
   context.getPipelineOptions());
@@ -570,8 +567,6 @@ class FlinkBatchTransformTranslators {
   DataSet> inputDataSet =
   context.getInputDataSet(context.getInput(transform));
 
-  final OldDoFn oldDoFn = transform.getFn();
-
   Map, PCollection> outputs = 
context.getOutput(transform).getAll();
 
   Map, Integer> outputMap = Maps.newHashMap();
@@ -618,7 +613,7 @@ class FlinkBatchTransformTranslators {
   @SuppressWarnings("unchecked")
   FlinkMultiOutputDoFnFunction doFnWrapper =
   new FlinkMultiOutputDoFnFunction(
-  oldDoFn,
+  doFn,
   windowingStrategy,
   sideInputStrategies,
   context.getPipelineOptions(),

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8330bfa7/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
index 042f8df..42ef630 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
@@ -358,7 +358,7 @@ public class FlinkStreamingTransformTra

[GitHub] incubator-beam pull request #1565: [BEAM-498] Practically removes OldDoFn fr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1565


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752696#comment-15752696
 ] 

ASF GitHub Bot commented on BEAM-498:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1565


> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2111

[Apache Jenkins Server]

See 

--
Started by user davor
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/masterv0.4.0-incubating-RC2^{commit} # timeout=10
 > git rev-parse masterv0.4.0-incubating-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/masterv0.4.0-incubating-RC2^{commit} # timeout=10
 > git rev-parse masterv0.4.0-incubating-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse origin/masterv0.4.0-incubating-RC2^{commit} # timeout=10
 > git rev-parse masterv0.4.0-incubating-RC2^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.

[jira] [Updated] (BEAM-1166) Source.getDefaultOutputCoder() documentation should mention CannotProvideCoderException

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1166:
--
Component/s: sdk-java-core
 beam-model

> Source.getDefaultOutputCoder() documentation should mention 
> CannotProvideCoderException
> ---
>
> Key: BEAM-1166
> URL: https://issues.apache.org/jira/browse/BEAM-1166
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> Knowing that you can throw CannotProviderCoderException is an important part 
> of implementing getDefaultOutputCoder
> The documentation for PTransform's getDefaultOutputCoder mentions this class, 
> and we should do this for the Source class as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1120) Prepare Dataflow runner for 0.4.0 release

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1120.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Prepare Dataflow runner for 0.4.0 release
> -
>
> Key: BEAM-1120
> URL: https://issues.apache.org/jira/browse/BEAM-1120
> Project: Beam
>  Issue Type: Improvement
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1125) Rename PTransform.apply to PTransform.expand

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752796#comment-15752796
 ] 

ASF GitHub Bot commented on BEAM-1125:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1634

[BEAM-1125] Rename PTransform.apply() to PTransform.expand()

Rename apply function to expand to match the recent change in the Java SDK.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam expand

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1634.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1634


commit b6a0974375e5c7414a52cd3ffd2a9e9fe8d1889f
Author: Ahmet Altay 
Date:   2016-12-15T22:27:08Z

Rename PTransform.apply() to PTransform.expand()




> Rename PTransform.apply to PTransform.expand
> 
>
> Key: BEAM-1125
> URL: https://issues.apache.org/jira/browse/BEAM-1125
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>  Labels: sdk-consistency
>
> For context see:
> [BEAM-438] https://issues.apache.org/jira/browse/BEAM-438
> [PR #1538] https://github.com/apache/incubator-beam/pull/1538
> https://lists.apache.org/thread.html/b4d9bcfbfeaa5dbcd5b68fd2344cdffe45587ff88cb714638504e759@%3Cdev.beam.apache.org%3E
> This requires renaming the apply method, updating all custom PTransforms, and 
> runners where transform.apply is called. (Based on the Java PR, this could be 
> easily done with a refactoring tool.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1634: [BEAM-1125] Rename PTransform.apply() to ...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1634

[BEAM-1125] Rename PTransform.apply() to PTransform.expand()

Rename apply function to expand to match the recent change in the Java SDK.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam expand

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1634.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1634


commit b6a0974375e5c7414a52cd3ffd2a9e9fe8d1889f
Author: Ahmet Altay 
Date:   2016-12-15T22:27:08Z

Rename PTransform.apply() to PTransform.expand()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #2112

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1167) Jobs that contain FileBasedSources with globs that expand into zero files should fail

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-1167:


 Summary: Jobs that contain FileBasedSources with globs that expand 
into zero files should fail
 Key: BEAM-1167
 URL: https://issues.apache.org/jira/browse/BEAM-1167
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core, sdk-py
Reporter: Chamikara Jayalath


Currently we do not fail if a glob that is used to create a FileBasedSource 
expand into zero files. This can be error prone since a typo when specifying a 
glob pattern can result in a job that silently pass without producing any 
output. I believe this should be fixed as follows.

If a glob pattern specified as input to a FileBasedSource expand into zero 
files.
(1) If validation of FileBasedSource is enabled, fail the job at pipeline 
construction.
(2) If validation is disabled, fail the job at runtime by raising an exception 
at initial splitting.

Both Java and Python SDKs have to be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #1625

[bchambers]

Closes #1625


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8c189137
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8c189137
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8c189137

Branch: refs/heads/master
Commit: 8c1891375a16bb13f65b78b4dd012330cbbfa85a
Parents: 5a3ace4 ce75ba1
Author: bchambers 
Authored: Thu Dec 15 15:50:09 2016 -0800
Committer: bchambers 
Committed: Thu Dec 15 15:50:09 2016 -0800

--
 .../main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Don't incorrectly log error in MetricsEnvironment

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5a3ace4a7 -> 8c1891375


Don't incorrectly log error in MetricsEnvironment

Using getCurrentContainer() logs an error if metrics are not supported.
This is because it acts as the common point of access for user code that
reports metrics.

It should not be used within setCurrentContainer(), because the first
container being set will have a null previous-current-container, which
will cause the error to be incorrectly logged.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ce75ba1e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ce75ba1e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ce75ba1e

Branch: refs/heads/master
Commit: ce75ba1ec5b4265bad9f3ff9b4b66818424b1375
Parents: 5a3ace4
Author: bchambers 
Authored: Wed Dec 14 11:23:39 2016 -0800
Committer: bchambers 
Committed: Thu Dec 15 15:50:08 2016 -0800

--
 .../main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ce75ba1e/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
index 7c06cbf..5d7cb0b 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsEnvironment.java
@@ -56,7 +56,7 @@ public class MetricsEnvironment {
*/
   @Nullable
   public static MetricsContainer setCurrentContainer(@Nullable 
MetricsContainer container) {
-MetricsContainer previous = getCurrentContainer();
+MetricsContainer previous = CONTAINER_FOR_THREAD.get();
 if (container == null) {
   CONTAINER_FOR_THREAD.remove();
 } else {

[GitHub] incubator-beam pull request #1625: Don't incorrectly log error in MetricsEnv...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1625


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1635: Remove deprecated methods of InMemoryTime...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1635

Remove deprecated methods of InMemoryTimerInternals

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @tweise just choosing reviewer at random
CC: @scwhittle for the deprecation which I think is now cleaned up

(no JIRA because it is just cleanup)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
remove-deprecated-TimerInternals

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1635.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1635


commit e26f4075af6f2c990e23dc9f8fc8be2233652a9f
Author: Kenneth Knowles 
Date:   2016-12-16T00:02:23Z

Remove deprecated methods of InMemoryTimerInternals




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752949#comment-15752949
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1636

[BEAM-498] Moves OldDoFn to runners-core

This will of course need the usual Dataflow worker surgery.

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam move-old-do-fn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1636.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1636


commit 374a450a66baf43786e2a22afb8e8832d3146441
Author: Eugene Kirpichov 
Date:   2016-12-16T00:16:46Z

Moves OldDoFn to runners-core




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2115

[Apache Jenkins Server]

See 

--
[...truncated 5829 lines...]
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.384 sec - in 
org.apache.beam.sdk.transforms.LatestTest
Running org.apache.beam.sdk.transforms.MapElementsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.17 sec - in 
org.apache.beam.sdk.transforms.MapElementsTest
Running org.apache.beam.sdk.transforms.WithTimestampsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.321 sec - in 
org.apache.beam.sdk.transforms.WithTimestampsTest
Running org.apache.beam.sdk.transforms.TopTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.643 sec - in 
org.apache.beam.sdk.transforms.TopTest
Running org.apache.beam.sdk.transforms.FlatMapElementsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046 sec - in 
org.apache.beam.sdk.transforms.FlatMapElementsTest
Running org.apache.beam.sdk.transforms.ApproximateUniqueTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.873 sec - in 
org.apache.beam.sdk.transforms.ApproximateUniqueTest
Running org.apache.beam.sdk.transforms.ViewTest
Tests run: 37, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.125 sec - in 
org.apache.beam.sdk.transforms.ViewTest
Running org.apache.beam.sdk.transforms.CombineFnsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.234 sec - in 
org.apache.beam.sdk.transforms.CombineFnsTest
Running org.apache.beam.sdk.transforms.GroupByKeyTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.28 sec - in 
org.apache.beam.sdk.transforms.GroupByKeyTest
Running org.apache.beam.sdk.transforms.FilterTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.26 sec - in 
org.apache.beam.sdk.transforms.FilterTest
Running org.apache.beam.sdk.transforms.ParDoLifecycleTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.388 sec - in 
org.apache.beam.sdk.transforms.ParDoLifecycleTest
Running org.apache.beam.sdk.transforms.ApproximateQuantilesTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.282 sec - in 
org.apache.beam.sdk.transforms.ApproximateQuantilesTest
Running org.apache.beam.sdk.transforms.join.CoGroupByKeyTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.285 sec - in 
org.apache.beam.sdk.transforms.join.CoGroupByKeyTest
Running org.apache.beam.sdk.transforms.DoFnTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.071 sec - in 
org.apache.beam.sdk.transforms.DoFnTest
Running org.apache.beam.sdk.transforms.windowing.WindowTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.051 sec - in 
org.apache.beam.sdk.transforms.windowing.WindowTest
Running org.apache.beam.sdk.transforms.windowing.WindowingTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.332 sec - in 
org.apache.beam.sdk.transforms.windowing.WindowingTest
Running org.apache.beam.sdk.transforms.ValuesTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.044 sec - in 
org.apache.beam.sdk.transforms.ValuesTest
Running org.apache.beam.sdk.transforms.SplittableDoFnTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.103 sec - in 
org.apache.beam.sdk.transforms.SplittableDoFnTest
Running org.apache.beam.sdk.transforms.RegexTest
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.36 sec - in 
org.apache.beam.sdk.transforms.RegexTest
Running org.apache.beam.sdk.transforms.PartitionTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.316 sec - in 
org.apache.beam.sdk.transforms.PartitionTest
Running org.apache.beam.sdk.transforms.ParDoTest
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.836 sec - in 
org.apache.beam.sdk.transforms.ParDoTest
Running org.apache.beam.sdk.transforms.CreateTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.DistinctTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.153 sec - in 
org.apache.beam.sdk.transforms.DistinctTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.046 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.transforms.KvSwapTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 sec - in 
org.apache.beam.sdk.transforms.KvSwapTest
Running org.apache.beam.sdk.util.ReshuffleTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.234 sec - in 
org.apache.beam.sdk.util.ReshuffleTest
Running org.apache.beam.runners.core.UnboundedReadFromBoundedSourceTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.302 sec - in 
org.apache.beam.runners.core.UnboundedReadFromBoundedSourceTest

Results :

Te

[GitHub] incubator-beam pull request #1636: [BEAM-498] Moves OldDoFn to runners-core

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1636

[BEAM-498] Moves OldDoFn to runners-core

This will of course need the usual Dataflow worker surgery.

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam move-old-do-fn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1636.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1636


commit 374a450a66baf43786e2a22afb8e8832d3146441
Author: Eugene Kirpichov 
Date:   2016-12-16T00:16:46Z

Moves OldDoFn to runners-core




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1637: [BEAM-1153] GcsUtil: use non-batch API fo...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1637

[BEAM-1153] GcsUtil: use non-batch API for single file size requests.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam release-0.4.0-incubating

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1637


commit 58601f8c69b97dbdd9087b27c356c50bca7a1c8b
Author: Pei He 
Date:   2016-12-14T02:29:17Z

[BEAM-1153] GcsUtil: use non-batch API for single file size requests.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1153) GcsUtil needs to set timeout and retry explicitly in BatchRequest.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752972#comment-15752972
 ] 

ASF GitHub Bot commented on BEAM-1153:
--

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1637

[BEAM-1153] GcsUtil: use non-batch API for single file size requests.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam release-0.4.0-incubating

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1637.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1637


commit 58601f8c69b97dbdd9087b27c356c50bca7a1c8b
Author: Pei He 
Date:   2016-12-14T02:29:17Z

[BEAM-1153] GcsUtil: use non-batch API for single file size requests.




> GcsUtil needs to set timeout and retry explicitly in BatchRequest.
> --
>
> Key: BEAM-1153
> URL: https://issues.apache.org/jira/browse/BEAM-1153
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Pei He
>Priority: Blocker
>
> Non-batch requests uses RetryHttpRequestInitializer, which set read timeout 
> as 80 seconds, and does more retries.
> Google Cloud auto generated Json library doesn't set HttpRequestInitializer 
> for batch requests.
> GcsUtil uses storageClient.batch(), and it is defined in here:
> https://github.com/vparfonov/google-api-java-client/blob/master/google-api-client/src/main/java/com/google/api/client/googleapis/services/AbstractGoogleClient.java#L256
> Without the HttpRequestInitializer, the default read timeout is 20 seconds.
> Possible fix is: https://github.com/apache/incubator-beam/pull/1608
> In additional, we can partially rollback 
> https://github.com/apache/incubator-beam/pull/1359 to keep using non-batch 
> API for fileSize() for single files. This will make sure existing code will 
> keep work as the same way.
> PR: https://github.com/apache/incubator-beam/pull/1611



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Update the BQ export flat from Json to Avro

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk e383c7715 -> d3c887480


Update the BQ export flat from Json to Avro


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0a558c71
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0a558c71
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0a558c71

Branch: refs/heads/python-sdk
Commit: 0a558c7171d6e4452d88ecffd16a024a19cbfc42
Parents: e383c77
Author: Sourabh Bajaj 
Authored: Wed Dec 14 11:44:46 2016 -0800
Committer: Sourabh Bajaj 
Committed: Wed Dec 14 11:44:46 2016 -0800

--
 sdks/python/apache_beam/runners/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0a558c71/sdks/python/apache_beam/runners/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow_runner.py
index 8b953b0..a3f7d94 100644
--- a/sdks/python/apache_beam/runners/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow_runner.py
@@ -520,7 +520,7 @@ class DataflowPipelineRunner(PipelineRunner):
 elif transform.source.format == 'text':
   step.add_property(PropertyNames.FILE_PATTERN, transform.source.path)
 elif transform.source.format == 'bigquery':
-  step.add_property(PropertyNames.BIGQUERY_EXPORT_FORMAT, 'FORMAT_JSON')
+  step.add_property(PropertyNames.BIGQUERY_EXPORT_FORMAT, 'FORMAT_AVRO')
   # TODO(silviuc): Add table validation if transform.source.validate.
   if transform.source.table_reference is not None:
 step.add_property(PropertyNames.BIGQUERY_DATASET,

[2/2] incubator-beam git commit: Closes #1617

[robertwb]

Closes #1617


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d3c88748
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d3c88748
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d3c88748

Branch: refs/heads/python-sdk
Commit: d3c88748099fcccb27aef67c5c390d0bc67ebeb0
Parents: e383c77 0a558c7
Author: Robert Bradshaw 
Authored: Thu Dec 15 16:35:59 2016 -0800
Committer: Robert Bradshaw 
Committed: Thu Dec 15 16:35:59 2016 -0800

--
 sdks/python/apache_beam/runners/dataflow_runner.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-964) Investing exporting BQ as Avro instead of Json for dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752986#comment-15752986
 ] 

ASF GitHub Bot commented on BEAM-964:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1617


> Investing exporting BQ as Avro instead of Json for dataflow runner
> --
>
> Key: BEAM-964
> URL: https://issues.apache.org/jira/browse/BEAM-964
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1617: [BEAM-964] json avro flag migration

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1617


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---