Jenkins build is back to stable : beam_PostCommit_MavenVerify #1931

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1052) UnboundedSource splitId uniqueness breaks if more than one source is used.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704575#comment-15704575
 ] 

ASF GitHub Bot commented on BEAM-1052:
--

GitHub user aviemzur opened a pull request:

https://github.com/apache/incubator-beam/pull/1450

[BEAM-1052] Add InputDStream id to MicrobatchSource hashcode.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Done to avoid collisions between splits of different sources.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/incubator-beam 
add-inputdstream-id-to-microbatchsource-hash-code

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1450.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1450






> UnboundedSource splitId uniqueness breaks if more than one source is used.
> --
>
> Key: BEAM-1052
> URL: https://issues.apache.org/jira/browse/BEAM-1052
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Aviem Zur
>
> We use a "running-id" to identify source splits, but we reiterate for each 
> source evaluated.
> Spark already assigns a unique id per InputDStream, it would be unique enough 
> if we replace {{MicrobatchSource}} hash code with one containing both the 
> running-id and the InputDStream id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1450: [BEAM-1052] Add InputDStream id to Microb...

[aviemzur]

GitHub user aviemzur opened a pull request:

https://github.com/apache/incubator-beam/pull/1450

[BEAM-1052] Add InputDStream id to MicrobatchSource hashcode.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Done to avoid collisions between splits of different sources.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/incubator-beam 
add-inputdstream-id-to-microbatchsource-hash-code

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1450.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1450






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1675

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-1052) UnboundedSource splitId uniqueness breaks if more than one source is used.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1052:

Description: 
We use a "running-id" to identify source splits, but we reiterate for each 
source evaluated.
Spark already assigns a unique id per InputDStream, it would be unique enough 
if we replace {{MicrobatchSource}} hash code with one containing both the 
running-id and the InputDStream id.

  was:
We use a "running-id" to identify source splits, but we reiterate for each 
source evaluated.
Spark already assigns a unique id per InputDStream, it would be unique enough 
if we replace MicrobatchSource hash code with one containing both the 
running-id and the InputDStream id.


> UnboundedSource splitId uniqueness breaks if more than one source is used.
> --
>
> Key: BEAM-1052
> URL: https://issues.apache.org/jira/browse/BEAM-1052
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Aviem Zur
>
> We use a "running-id" to identify source splits, but we reiterate for each 
> source evaluated.
> Spark already assigns a unique id per InputDStream, it would be unique enough 
> if we replace {{MicrobatchSource}} hash code with one containing both the 
> running-id and the InputDStream id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1052) UnboundedSource splitId uniqueness breaks if more than one source is used.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1052:

Description: 
We use a "running-id" to identify source splits, but we reiterate for each 
source evaluated.
Spark already assigns a unique id per InputDStream, it would be unique enough 
if we replace MicrobatchSource hash code with one containing both the 
running-id and the InputDStream id.

  was:
We use a "running-id" to identify source splits, but we reiterate for each 
source evaluated.
Spark already assigns a unique id per InputDStream, it would be unique enough 
if we replace ```MicrobatchSource``` hash code with one containing both the 
running-id and the InputDStream id.


> UnboundedSource splitId uniqueness breaks if more than one source is used.
> --
>
> Key: BEAM-1052
> URL: https://issues.apache.org/jira/browse/BEAM-1052
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Aviem Zur
>
> We use a "running-id" to identify source splits, but we reiterate for each 
> source evaluated.
> Spark already assigns a unique id per InputDStream, it would be unique enough 
> if we replace MicrobatchSource hash code with one containing both the 
> running-id and the InputDStream id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1052) UnboundedSource splitId uniqueness breaks if more than one source is used.

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1052:

Description: 
We use a "running-id" to identify source splits, but we reiterate for each 
source evaluated.
Spark already assigns a unique id per InputDStream, it would be unique enough 
if we replace ```MicrobatchSource``` hash code with one containing both the 
running-id and the InputDStream id.

  was:
We use a "running-id" to identify source splits, but we reiterate for each 
source evaluated.
Spark already assigns a unique id per InputDStream, it would be unique enough 
if we replace the int ids with a String such as: InputDStream.id() + "_" + 
running-id.


> UnboundedSource splitId uniqueness breaks if more than one source is used.
> --
>
> Key: BEAM-1052
> URL: https://issues.apache.org/jira/browse/BEAM-1052
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Aviem Zur
>
> We use a "running-id" to identify source splits, but we reiterate for each 
> source evaluated.
> Spark already assigns a unique id per InputDStream, it would be unique enough 
> if we replace ```MicrobatchSource``` hash code with one containing both the 
> running-id and the InputDStream id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1387: [BEAM-1006] Fixing the splitKeysToFilters...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1387


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1006) splitKeysToFilter method is creating a bad query string.

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-1006.

Resolution: Fixed

> splitKeysToFilter method is creating a bad query string.
> 
>
> Key: BEAM-1006
> URL: https://issues.apache.org/jira/browse/BEAM-1006
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.3.0-incubating
>Reporter: David Billings
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>  Labels: mongodb
> Fix For: 0.4.0-incubating
>
>
> When you have a large number of documents in your mongo collection the 
> splitKeysToFilter method is invoked. 
> Currently Document{{_id= is prefixed to the query string
> {code}{ $and: [ 
> {"_id":{$lte:Objectd("Document{{_id=564a0ad6e21b840001d75f5d}}")}} ]}{code}
> This causes the document parse to fail with the following exception: 
> {code}Exception in thread "main" java.lang.IllegalArgumentException: invalid 
> hexadecimal representation of an ObjectId: 
> [Document{{_id=56e1ec546b2f0700015e61b2}}]{code}
> This is caused by line 305 
> {code}
> String splitKey = splitKeys.get(i).toString();
> {code}
> The code should use 
> {code}
> String splitKey = splitKeys.get(i).get("_id").toString();
> {code}
> to just return the ID. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1006) splitKeysToFilter method is creating a bad query string.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704476#comment-15704476
 ] 

ASF GitHub Bot commented on BEAM-1006:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1387


> splitKeysToFilter method is creating a bad query string.
> 
>
> Key: BEAM-1006
> URL: https://issues.apache.org/jira/browse/BEAM-1006
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.3.0-incubating
>Reporter: David Billings
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>  Labels: mongodb
> Fix For: 0.4.0-incubating
>
>
> When you have a large number of documents in your mongo collection the 
> splitKeysToFilter method is invoked. 
> Currently Document{{_id= is prefixed to the query string
> {code}{ $and: [ 
> {"_id":{$lte:Objectd("Document{{_id=564a0ad6e21b840001d75f5d}}")}} ]}{code}
> This causes the document parse to fail with the following exception: 
> {code}Exception in thread "main" java.lang.IllegalArgumentException: invalid 
> hexadecimal representation of an ObjectId: 
> [Document{{_id=56e1ec546b2f0700015e61b2}}]{code}
> This is caused by line 305 
> {code}
> String splitKey = splitKeys.get(i).toString();
> {code}
> The code should use 
> {code}
> String splitKey = splitKeys.get(i).get("_id").toString();
> {code}
> to just return the ID. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: [BEAM-1006] This closes #1387

[jbonofre]

[BEAM-1006] This closes #1387


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3f16f266
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3f16f266
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3f16f266

Branch: refs/heads/master
Commit: 3f16f26600ac6e552f379c94cfae1bc57ed5f5f0
Parents: aeff1d5 b453457
Author: Jean-Baptiste Onofré 
Authored: Tue Nov 29 08:15:08 2016 +0100
Committer: Jean-Baptiste Onofré 
Committed: Tue Nov 29 08:15:08 2016 +0100

--
 .../main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java| 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--

[1/2] incubator-beam git commit: [BEAM-1006] Fixing the MongoDbIO splitKeysToFilters, calling to string on the BSON document prefixed Document to the string, updated it to just get the value of ID

[jbonofre]

Repository: incubator-beam
Updated Branches:
  refs/heads/master aeff1d5c2 -> 3f16f2660


[BEAM-1006] Fixing the MongoDbIO splitKeysToFilters, calling to string on the 
BSON document prefixed Document to the string, updated it to just get the value 
of ID


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b453457f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b453457f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b453457f

Branch: refs/heads/master
Commit: b453457f5699963f9a72129ab86ef18b71c04e61
Parents: aeff1d5
Author: DavidB 
Authored: Fri Nov 18 15:11:47 2016 +
Committer: Jean-Baptiste Onofré 
Committed: Tue Nov 29 07:50:40 2016 +0100

--
 .../main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java| 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b453457f/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
--
diff --git 
a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
 
b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
index cd5cdaf..70239e6 100644
--- 
a/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
+++ 
b/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java
@@ -302,12 +302,12 @@ public class MongoDbIO {
   ArrayList filters = new ArrayList<>();
   String lowestBound = null; // lower boundary (previous split in the 
iteration)
   for (int i = 0; i < splitKeys.size(); i++) {
-String splitKey = splitKeys.get(i).toString();
-String rangeFilter = null;
+String splitKey = splitKeys.get(i).get("_id").toString();
+String rangeFilter;
 if (i == 0) {
   // this is the first split in the list, the filter defines
   // the range from the beginning up to this split
-  rangeFilter = String.format("{ $and: [ 
{\"_id\":{$lte:Objectd(\"%s\")}}",
+  rangeFilter = String.format("{ $and: [ 
{\"_id\":{$lte:ObjectId(\"%s\")}}",
   splitKey);
 } else if (i == splitKeys.size() - 1) {
   // this is the last split in the list, the filter defines

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1674

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify #1930

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1673

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1449: Demonstrate serializing issue

[dhalperi]

Github user dhalperi closed the pull request at:

https://github.com/apache/incubator-beam/pull/1449


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1449: Demonstrate serializing issue

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1449

Demonstrate serializing issue

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/nvp_repro

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1449.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1449


commit c9a5f8d8bfbe48921b1327acb4d2bfe8fa31f12e
Author: Sam McVeety 
Date:   2016-11-02T00:58:16Z

Demonstrate serializing issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704249#comment-15704249
 ] 

ASF GitHub Bot commented on BEAM-551:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1230


> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Demonstrate PubsubIO with NVP

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ae06f759f -> aeff1d5c2


Demonstrate PubsubIO with NVP


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f9225981
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f9225981
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f9225981

Branch: refs/heads/master
Commit: f92259814964fb4d3b2381187247b3f11b5fe33f
Parents: ae06f75
Author: Sam McVeety 
Authored: Sat Oct 29 19:02:51 2016 -0700
Committer: Dan Halperin 
Committed: Mon Nov 28 21:14:33 2016 -0800

--
 .../java/org/apache/beam/sdk/io/PubsubIO.java   | 176 ---
 .../apache/beam/sdk/io/PubsubUnboundedSink.java |  23 ++-
 .../beam/sdk/io/PubsubUnboundedSource.java  |  40 +++--
 .../org/apache/beam/sdk/io/PubsubIOTest.java|  43 +++--
 .../beam/sdk/io/PubsubUnboundedSinkTest.java|  20 ++-
 .../beam/sdk/io/PubsubUnboundedSourceTest.java  |  14 +-
 6 files changed, 232 insertions(+), 84 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f9225981/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
index 72a6399..9768788 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
@@ -31,11 +31,15 @@ import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.coders.VoidCoder;
 import org.apache.beam.sdk.options.PubsubOptions;
+import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
 import org.apache.beam.sdk.util.CoderUtils;
@@ -134,7 +138,7 @@ public class PubsubIO {
* Populate common {@link DisplayData} between Pubsub source and sink.
*/
   private static void populateCommonDisplayData(DisplayData.Builder builder,
-  String timestampLabel, String idLabel, PubsubTopic topic) {
+  String timestampLabel, String idLabel, String topic) {
 builder
 .addIfNotNull(DisplayData.item("timestampLabel", timestampLabel)
 .withLabel("Timestamp Label Attribute"))
@@ -142,7 +146,7 @@ public class PubsubIO {
 .withLabel("ID Label Attribute"));
 
 if (topic != null) {
-  builder.add(DisplayData.item("topic", topic.asPath())
+  builder.add(DisplayData.item("topic", topic)
   .withLabel("Pubsub Topic"));
 }
   }
@@ -253,6 +257,61 @@ public class PubsubIO {
   }
 
   /**
+   * Used to build a {@link ValueProvider} for {@link PubsubSubscription}.
+   */
+  private static class SubscriptionTranslator
+  implements SerializableFunction {
+@Override
+public PubsubSubscription apply(String from) {
+  return PubsubSubscription.fromPath(from);
+}
+  }
+
+  /**
+   * Used to build a {@link ValueProvider} for {@link SubscriptionPath}.
+   */
+  private static class SubscriptionPathTranslator
+  implements SerializableFunction {
+@Override
+public SubscriptionPath apply(PubsubSubscription from) {
+  return PubsubClient.subscriptionPathFromName(from.project, 
from.subscription);
+}
+  }
+
+  /**
+   * Used to build a {@link ValueProvider} for {@link PubsubTopic}.
+   */
+  private static class TopicTranslator
+  implements SerializableFunction {
+@Override
+public PubsubTopic apply(String from) {
+  return PubsubTopic.fromPath(from);
+}
+  }
+
+  /**
+   * Used to build a {@link ValueProvider} for {@link TopicPath}.
+   */
+  private static class TopicPathTranslator
+  implements SerializableFunction {
+@Override
+public TopicPath apply(PubsubTopic from) {
+  return PubsubClient.topicPathFromName(from.project, from.topic);
+}
+  }
+
+  /**
+   * Used to build a {@link ValueProvider} for {@link ProjectPath}.
+   */
+  private static class ProjectPathTranslator
+  implements 

[GitHub] incubator-beam pull request #1230: [BEAM-551] PubsubIO with ValueProvider

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1230


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1230

[dhalperi]

Closes #1230


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/aeff1d5c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/aeff1d5c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/aeff1d5c

Branch: refs/heads/master
Commit: aeff1d5c219385cce20a275a4e47d9184f5cf59c
Parents: ae06f75 f922598
Author: Dan Halperin 
Authored: Mon Nov 28 21:21:35 2016 -0800
Committer: Dan Halperin 
Committed: Mon Nov 28 21:21:35 2016 -0800

--
 .../java/org/apache/beam/sdk/io/PubsubIO.java   | 176 ---
 .../apache/beam/sdk/io/PubsubUnboundedSink.java |  23 ++-
 .../beam/sdk/io/PubsubUnboundedSource.java  |  40 +++--
 .../org/apache/beam/sdk/io/PubsubIOTest.java|  43 +++--
 .../beam/sdk/io/PubsubUnboundedSinkTest.java|  20 ++-
 .../beam/sdk/io/PubsubUnboundedSourceTest.java  |  14 +-
 6 files changed, 232 insertions(+), 84 deletions(-)
--

[1/2] incubator-beam git commit: Add method to output runtime options

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master cdb7ba165 -> ae06f759f


Add method to output runtime options


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ee52318f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ee52318f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ee52318f

Branch: refs/heads/master
Commit: ee52318f2512c6661823e4f546f84dbc2caa955b
Parents: cdb7ba1
Author: sammcveety 
Authored: Fri Oct 21 12:50:01 2016 -0400
Committer: Dan Halperin 
Committed: Mon Nov 28 20:24:38 2016 -0800

--
 .../beam/sdk/options/PipelineOptions.java   |  7 ++
 .../sdk/options/PipelineOptionsFactory.java |  1 +
 .../sdk/options/ProxyInvocationHandler.java | 26 
 .../beam/sdk/options/PipelineOptionsTest.java   | 24 ++
 4 files changed, 58 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ee52318f/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
index 2139ed9..ddb040d 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
@@ -24,6 +24,7 @@ import 
com.fasterxml.jackson.databind.annotation.JsonSerialize;
 import com.google.auto.service.AutoService;
 import com.google.common.base.MoreObjects;
 import java.lang.reflect.Proxy;
+import java.util.Map;
 import java.util.ServiceLoader;
 import java.util.concurrent.ThreadLocalRandom;
 import java.util.concurrent.atomic.AtomicLong;
@@ -322,6 +323,12 @@ public interface PipelineOptions extends HasDisplayData {
   }
 
   /**
+   * Returns a map of properties which correspond to {@link 
ValueProvider.RuntimeValueProvider},
+   * keyed by the property name.  The value is a map containing type and 
default information.
+   */
+  Map> outputRuntimeOptions();
+
+  /**
* Provides a unique ID for this {@link PipelineOptions} object, assigned at 
graph
* construction time.
*/

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ee52318f/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index 6009867..9805489 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -1219,6 +1219,7 @@ public class PipelineOptionsFactory {
 // Ignore methods on the base PipelineOptions interface.
 try {
   knownMethods.add(iface.getMethod("as", Class.class));
+  knownMethods.add(iface.getMethod("outputRuntimeOptions"));
   knownMethods.add(iface.getMethod("populateDisplayData", 
DisplayData.Builder.class));
 } catch (NoSuchMethodException | SecurityException e) {
   throw new RuntimeException(e);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ee52318f/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
index 86f9918..a0e3ec2 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
@@ -48,6 +48,7 @@ import java.io.IOException;
 import java.lang.annotation.Annotation;
 import java.lang.reflect.InvocationHandler;
 import java.lang.reflect.Method;
+import java.lang.reflect.ParameterizedType;
 import java.lang.reflect.Proxy;
 import java.lang.reflect.Type;
 import java.util.Arrays;
@@ -130,6 +131,8 @@ class ProxyInvocationHandler implements InvocationHandler {
   return equals(args[0]);
 } else if (args == null && "hashCode".equals(method.getName())) {
   return hashCode();
+} else if (args == null && 
"outputRuntimeOptions".equals(method.getName())) {
+  return outputRuntimeOptions((PipelineOptions) proxy);
 } else if (args != null && "as".equals(method.getName()) && args[0] 

[GitHub] incubator-beam pull request #1156: [BEAM-551] Add method to output runtime o...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1156


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704186#comment-15704186
 ] 

ASF GitHub Bot commented on BEAM-551:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1156


> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #1156

[dhalperi]

Closes #1156


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ae06f759
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ae06f759
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ae06f759

Branch: refs/heads/master
Commit: ae06f759fbf72fa31e02dc943ab46afe03471904
Parents: cdb7ba1 ee52318
Author: Dan Halperin 
Authored: Mon Nov 28 20:51:48 2016 -0800
Committer: Dan Halperin 
Committed: Mon Nov 28 20:51:48 2016 -0800

--
 .../beam/sdk/options/PipelineOptions.java   |  7 ++
 .../sdk/options/PipelineOptionsFactory.java |  1 +
 .../sdk/options/ProxyInvocationHandler.java | 26 
 .../beam/sdk/options/PipelineOptionsTest.java   | 24 ++
 4 files changed, 58 insertions(+)
--

[jira] [Resolved] (BEAM-1058) mvn clean verify is not tested

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1058.
---
   Resolution: Incomplete
Fix Version/s: (was: 0.4.0-incubating)
   Not applicable

> mvn clean verify is not tested
> --
>
> Key: BEAM-1058
> URL: https://issues.apache.org/jira/browse/BEAM-1058
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Daniel Halperin
>Assignee: Davor Bonaci
>Priority: Critical
> Fix For: Not applicable
>
>
> A simple {{mvn clean verify}} does not seem to be tested anywhere. It 
> currently fails at HEAD due to dependency issues, but this is not caught by 
> any of our test suites.
> Note that {{beam_PostCommit_MavenVerify}} actually runs {{install}} for some 
> reason.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1058) mvn clean verify is not tested

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704168#comment-15704168
 ] 

Daniel Halperin commented on BEAM-1058:
---

Temporarily closing -- the issue I ran into at HEAD may have been a local 
environmental problem from a lack of recent rebase.

> mvn clean verify is not tested
> --
>
> Key: BEAM-1058
> URL: https://issues.apache.org/jira/browse/BEAM-1058
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Daniel Halperin
>Assignee: Davor Bonaci
>Priority: Critical
> Fix For: 0.4.0-incubating
>
>
> A simple {{mvn clean verify}} does not seem to be tested anywhere. It 
> currently fails at HEAD due to dependency issues, but this is not caught by 
> any of our test suites.
> Note that {{beam_PostCommit_MavenVerify}} actually runs {{install}} for some 
> reason.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1058) mvn clean verify is not tested

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1058:
-

 Summary: mvn clean verify is not tested
 Key: BEAM-1058
 URL: https://issues.apache.org/jira/browse/BEAM-1058
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Daniel Halperin
Assignee: Davor Bonaci
Priority: Critical
 Fix For: 0.4.0-incubating


A simple {{mvn clean verify}} does not seem to be tested anywhere. It currently 
fails at HEAD due to dependency issues, but this is not caught by any of our 
test suites.

Note that {{beam_PostCommit_MavenVerify}} actually runs {{install}} for some 
reason.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1672

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1222: [Ignore, PR for discussion] Replace IOCha...

[peihe]

Github user peihe closed the pull request at:

https://github.com/apache/incubator-beam/pull/1222


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Remove the Experimental Annotation from the DirectRunner

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4bcef03d9 -> cdb7ba165


Remove the Experimental Annotation from the DirectRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0ae18124
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0ae18124
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0ae18124

Branch: refs/heads/master
Commit: 0ae181248823b08803a8f8ac0d39a35a190c4acd
Parents: 4bcef03
Author: Thomas Groh 
Authored: Mon Nov 28 15:52:03 2016 -0800
Committer: Thomas Groh 
Committed: Mon Nov 28 15:52:51 2016 -0800

--
 .../main/java/org/apache/beam/runners/direct/DirectRunner.java  | 5 +
 1 file changed, 1 insertion(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0ae18124/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index cb31947..f71e109 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -41,7 +41,6 @@ import org.apache.beam.sdk.AggregatorValues;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.Pipeline.PipelineExecutionException;
 import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.io.Write;
 import org.apache.beam.sdk.metrics.MetricResults;
@@ -71,9 +70,7 @@ import org.joda.time.Instant;
  * An In-Memory implementation of the Dataflow Programming Model. Supports 
Unbounded
  * {@link PCollection PCollections}.
  */
-@Experimental
-public class DirectRunner
-extends PipelineRunner {
+public class DirectRunner extends PipelineRunner {
   /**
* The default set of transform overrides to use in the {@link DirectRunner}.
*

[jira] [Commented] (BEAM-1032) Starter archetype pipeline has no runner on its classpath

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703831#comment-15703831
 ] 

Davor Bonaci commented on BEAM-1032:


Thanks [~swegner].

Fixed?

> Starter archetype pipeline has no runner on its classpath
> -
>
> Key: BEAM-1032
> URL: https://issues.apache.org/jira/browse/BEAM-1032
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> When generating a new maven project using the starter archetype, the pipeline 
> does not execute because it doesn't have a runner jar on it's classpath:
> $ mvn exec:java -Dexec.mainClass=swegner.StarterPipeline
> ...
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java 
> (default-cli) on project yo: An exception occured while executing the Java 
> class. null: InvocationTargetException: No Runner was specified and the 
> DirectRunner was not found on the classpath.
> In the examples archetype, we have the direct runner as a dependency. We 
> should do the same for the starter archetype.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Improve the speed of getting file sizes

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 1530a1727 -> ad4dc87a4


Improve the speed of getting file sizes


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7a059d37
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7a059d37
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7a059d37

Branch: refs/heads/python-sdk
Commit: 7a059d37e71b62702e8cdeafec6956fc7e1e38c4
Parents: 1530a17
Author: Sourabh Bajaj 
Authored: Mon Nov 21 15:50:21 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 28 17:40:37 2016 -0800

--
 sdks/python/apache_beam/io/filebasedsource.py | 42 ++
 sdks/python/apache_beam/io/fileio.py  | 31 
 sdks/python/apache_beam/io/fileio_test.py | 41 +
 sdks/python/apache_beam/io/gcsio.py   | 25 +
 sdks/python/apache_beam/io/gcsio_test.py  | 38 
 5 files changed, 163 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7a059d37/sdks/python/apache_beam/io/filebasedsource.py
--
diff --git a/sdks/python/apache_beam/io/filebasedsource.py 
b/sdks/python/apache_beam/io/filebasedsource.py
index 7d8f686..14eaf27 100644
--- a/sdks/python/apache_beam/io/filebasedsource.py
+++ b/sdks/python/apache_beam/io/filebasedsource.py
@@ -107,7 +107,8 @@ class FileBasedSource(iobase.BoundedSource):
 if self._concat_source is None:
   single_file_sources = []
   file_names = [f for f in fileio.ChannelFactory.glob(self._pattern)]
-  sizes = FileBasedSource._estimate_sizes_in_parallel(file_names)
+  sizes = FileBasedSource._estimate_sizes_of_files(file_names,
+   self._pattern)
 
   # We create a reference for FileBasedSource that will be serialized along
   # with each _SingleFileSource. To prevent this FileBasedSource from 
having
@@ -144,22 +145,32 @@ class FileBasedSource(iobase.BoundedSource):
 compression_type=self._compression_type)
 
   @staticmethod
-  def _estimate_sizes_in_parallel(file_names):
+  def _estimate_sizes_of_files(file_names, pattern=None):
+"""Returns the size of all the files as an ordered list based on the file
+names that are provided here. If the pattern is specified here then we use
+the size_of_files_in_glob method to get the size of files matching the glob
+for performance improvements instead of getting the size one by one.
+"""
 if not file_names:
   return []
 elif len(file_names) == 1:
   return [fileio.ChannelFactory.size_in_bytes(file_names[0])]
 else:
-  # ThreadPool crashes in old versions of Python (< 2.7.5) if created from 
a
-  # child thread. (http://bugs.python.org/issue10015)
-  if not hasattr(threading.current_thread(), '_children'):
-threading.current_thread()._children = weakref.WeakKeyDictionary()
-  pool = ThreadPool(
-  min(MAX_NUM_THREADS_FOR_SIZE_ESTIMATION, len(file_names)))
-  try:
-return pool.map(fileio.ChannelFactory.size_in_bytes, file_names)
-  finally:
-pool.terminate()
+  if pattern is None:
+# ThreadPool crashes in old versions of Python (< 2.7.5) if created
+# from a child thread. (http://bugs.python.org/issue10015)
+if not hasattr(threading.current_thread(), '_children'):
+  threading.current_thread()._children = weakref.WeakKeyDictionary()
+pool = ThreadPool(
+min(MAX_NUM_THREADS_FOR_SIZE_ESTIMATION, len(file_names)))
+try:
+  return pool.map(fileio.ChannelFactory.size_in_bytes, file_names)
+finally:
+  pool.terminate()
+  else:
+file_sizes = fileio.ChannelFactory.size_of_files_in_glob(pattern,
+ file_names)
+return [file_sizes[f] for f in file_names]
 
   def _validate(self):
 """Validate if there are actual files in the specified glob pattern
@@ -179,7 +190,10 @@ class FileBasedSource(iobase.BoundedSource):
 file_names = [f for f in fileio.ChannelFactory.glob(self._pattern)]
 if (len(file_names) <=
 FileBasedSource.MIN_NUMBER_OF_FILES_TO_STAT):
-  return sum(self._estimate_sizes_in_parallel(file_names))
+  # We're reading very few files so we can pass names without pattern
+  # as otherwise we'll try to do optimization based on the pattern and
+  # might end up reading much more data than needed for a few files.
+  return sum(self._estimate_sizes_of_files(file_names))
 else:
   # Estimating size of a random sample.
   # 

[2/2] incubator-beam git commit: This closes #1404

[davor]

This closes #1404


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ad4dc87a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ad4dc87a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ad4dc87a

Branch: refs/heads/python-sdk
Commit: ad4dc87a472387b507545ab80dbd2fe42e02cea3
Parents: 1530a17 7a059d3
Author: Davor Bonaci 
Authored: Mon Nov 28 17:40:50 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 28 17:40:50 2016 -0800

--
 sdks/python/apache_beam/io/filebasedsource.py | 42 ++
 sdks/python/apache_beam/io/fileio.py  | 31 
 sdks/python/apache_beam/io/fileio_test.py | 41 +
 sdks/python/apache_beam/io/gcsio.py   | 25 +
 sdks/python/apache_beam/io/gcsio_test.py  | 38 
 5 files changed, 163 insertions(+), 14 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1671

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-278) Add I/O section

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703771#comment-15703771
 ] 

ASF GitHub Bot commented on BEAM-278:
-

GitHub user melap opened a pull request:

https://github.com/apache/incubator-beam-site/pull/96

[BEAM-278] Add I/O section to programming guide

Also add a small Running pipeline section.

R: @davorbonaci

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/melap/incubator-beam-site progguide-io

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/96.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #96


commit 503954c41b87236f75bd29627296d0a5b9a60141
Author: melissa 
Date:   2016-11-29T01:05:32Z

[BEAM-278] Add I/O section to programming guide




> Add I/O section
> ---
>
> Key: BEAM-278
> URL: https://issues.apache.org/jira/browse/BEAM-278
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Devin Donnelly
>Assignee: Melissa Pashniak
>
> Document I/O Sources and Sinks APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #96: [BEAM-278] Add I/O section to programm...

[melap]

GitHub user melap opened a pull request:

https://github.com/apache/incubator-beam-site/pull/96

[BEAM-278] Add I/O section to programming guide

Also add a small Running pipeline section.

R: @davorbonaci

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/melap/incubator-beam-site progguide-io

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/96.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #96


commit 503954c41b87236f75bd29627296d0a5b9a60141
Author: melissa 
Date:   2016-11-29T01:05:32Z

[BEAM-278] Add I/O section to programming guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1009) Upgrade from mockito-all 1 to mockito-core 2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703751#comment-15703751
 ] 

ASF GitHub Bot commented on BEAM-1009:
--

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1448

[BEAM-1009] Update to Mockito 2 with mockito-core module.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam mockito-2-upgrade

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1448.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1448


commit cc7cc5a516eb213873d9a286c3da97a91f3b4125
Author: Pei He 
Date:   2016-11-28T22:11:32Z

[BEAM-1009] Update to Mockito 2 with mockito-core module.




> Upgrade from mockito-all 1 to mockito-core 2
> 
>
> Key: BEAM-1009
> URL: https://issues.apache.org/jira/browse/BEAM-1009
> Project: Beam
>  Issue Type: Test
>Reporter: Pei He
>Assignee: Pei He
>
> Mockito 2 provides useful features, and the mockito-all module is no longer 
> generated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1448: [BEAM-1009] Update to Mockito 2 with mock...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1448

[BEAM-1009] Update to Mockito 2 with mockito-core module.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam mockito-2-upgrade

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1448.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1448


commit cc7cc5a516eb213873d9a286c3da97a91f3b4125
Author: Pei He 
Date:   2016-11-28T22:11:32Z

[BEAM-1009] Update to Mockito 2 with mockito-core module.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1041) IOChannelFactory: supports for recursively remove().

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703677#comment-15703677
 ] 

Daniel Halperin commented on BEAM-1041:
---

Can this be done entirely IOCF-independently in IOChannelUtils?

> IOChannelFactory: supports for recursively remove().
> 
>
> Key: BEAM-1041
> URL: https://issues.apache.org/jira/browse/BEAM-1041
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Tenghuan He
>
> 1. FileIOChannelFactory:
> Maybe using deleteDirectory() as suggested in here:
> http://stackoverflow.com/questions/779519/delete-directories-recursively-in-java
> 2. GcsIOChannelFactory:
> Since gcs doesn't have directories, remove recursively means remove all files 
> whose prefixes match.
> However, there are two issues to consider:
> First. remove("gs://mybucket/directory-to-remove")
> a. it should remove "gs://mybucket/directory-to-remove/my-file".
> b. it should not remove "gs://mybucket/directory-to-remove-my-file".
> Second, there should be a way to skip the prefix match for 
> remove(collection-of-a-thousand-files), perhaps by adding a recursive boolean 
> parameter to remove().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1670

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1021) DatastoreIO for python

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703611#comment-15703611
 ] 

ASF GitHub Bot commented on BEAM-1021:
--

Github user vikkyrk closed the pull request at:

https://github.com/apache/incubator-beam/pull/1433


> DatastoreIO for python
> --
>
> Key: BEAM-1021
> URL: https://issues.apache.org/jira/browse/BEAM-1021
> Project: Beam
>  Issue Type: New Feature
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: datastoreio write/delete ptransform

[dhalperi]

datastoreio write/delete ptransform

update datastore_wordcount example to include writes


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d46203b7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d46203b7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d46203b7

Branch: refs/heads/python-sdk
Commit: d46203b7fcdc9895c9cee1d82710f48aba31a748
Parents: 3dbeb8e
Author: Vikas Kedigehalli 
Authored: Wed Nov 23 14:09:09 2016 -0800
Committer: Dan Halperin 
Committed: Mon Nov 28 15:54:27 2016 -0800

--
 .../apache_beam/examples/datastore_wordcount.py | 137 +++
 .../apache_beam/io/datastore/v1/datastoreio.py  | 104 +-
 .../io/datastore/v1/datastoreio_test.py |  46 +++
 .../io/datastore/v1/fake_datastore.py   |  17 +++
 .../apache_beam/io/datastore/v1/helper.py   |  35 -
 .../apache_beam/io/datastore/v1/helper_test.py  |  36 +
 .../io/datastore/v1/query_splitter.py   |   7 +-
 7 files changed, 349 insertions(+), 33 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d46203b7/sdks/python/apache_beam/examples/datastore_wordcount.py
--
diff --git a/sdks/python/apache_beam/examples/datastore_wordcount.py 
b/sdks/python/apache_beam/examples/datastore_wordcount.py
index af75b1c..6b9779b 100644
--- a/sdks/python/apache_beam/examples/datastore_wordcount.py
+++ b/sdks/python/apache_beam/examples/datastore_wordcount.py
@@ -22,14 +22,18 @@ from __future__ import absolute_import
 import argparse
 import logging
 import re
+import uuid
+
+from google.datastore.v1 import entity_pb2
+from google.datastore.v1 import query_pb2
+from googledatastore import helper as datastore_helper, PropertyFilter
 
 import apache_beam as beam
 from apache_beam.io.datastore.v1.datastoreio import ReadFromDatastore
+from apache_beam.io.datastore.v1.datastoreio import WriteToDatastore
 from apache_beam.utils.options import GoogleCloudOptions
 from apache_beam.utils.options import PipelineOptions
 from apache_beam.utils.options import SetupOptions
-from google.datastore.v1 import query_pb2
-
 
 empty_line_aggregator = beam.Aggregator('emptyLines')
 average_word_size_aggregator = beam.Aggregator('averageWordLength',
@@ -41,7 +45,7 @@ class WordExtractingDoFn(beam.DoFn):
   """Parse each line of input text into words."""
 
   def process(self, context):
-"""Returns an iterator over the words of this element.
+"""Returns an iterator over words in contents of Cloud Datastore entity.
 The element is a line of text.  If the line is blank, note that, too.
 Args:
   context: the call-specific context: data and aggregator.
@@ -61,10 +65,100 @@ class WordExtractingDoFn(beam.DoFn):
 return words
 
 
+class EntityWrapper(object):
+  """Create a Cloud Datastore entity from the given string."""
+  def __init__(self, namespace, kind, ancestor):
+self._namespace = namespace
+self._kind = kind
+self._ancestor = ancestor
+
+  def make_entity(self, content):
+entity = entity_pb2.Entity()
+if self._namespace is not None:
+  entity.key.partition_id.namespace_id = self._namespace
+
+# All entities created will have the same ancestor
+datastore_helper.add_key_path(entity.key, self._kind, self._ancestor,
+  self._kind, str(uuid.uuid4()))
+
+datastore_helper.add_properties(entity, {"content": unicode(content)})
+return entity
+
+
+def write_to_datastore(project, user_options, pipeline_options):
+  """Creates a pipeline that writes entities to Cloud Datastore."""
+  p = beam.Pipeline(options=pipeline_options)
+
+  # pylint: disable=expression-not-assigned
+  (p
+   | 'read' >> beam.io.Read(beam.io.TextFileSource(user_options.input))
+   | 'create entity' >> beam.Map(
+   EntityWrapper(user_options.namespace, user_options.kind,
+ user_options.ancestor).make_entity)
+   | 'write to datastore' >> WriteToDatastore(project))
+
+  # Actually run the pipeline (all operations above are deferred).
+  p.run()
+
+
+def make_ancestor_query(kind, namespace, ancestor):
+  """Creates a Cloud Datastore ancestor query.
+
+  The returned query will fetch all the entities that have the parent key name
+  set to the given `ancestor`.
+  """
+  ancestor_key = entity_pb2.Key()
+  datastore_helper.add_key_path(ancestor_key, kind, ancestor)
+  if namespace is not None:
+ancestor_key.partition_id.namespace_id = namespace
+
+  query = query_pb2.Query()
+  query.kind.add().name = kind
+
+  datastore_helper.set_property_filter(
+  query.filter, '__key__', PropertyFilter.HAS_ANCESTOR, ancestor_key)
+
+  return query
+
+
+def 

[GitHub] incubator-beam pull request #1447: Remove the Experimental Annotation from t...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1447

Remove the Experimental Annotation from the DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
not_experimental_direct_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1447.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1447


commit 0ae181248823b08803a8f8ac0d39a35a190c4acd
Author: Thomas Groh 
Date:   2016-11-28T23:52:03Z

Remove the Experimental Annotation from the DirectRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1433: [BEAM-1021] python datastoreio write/dele...

[vikkyrk]

Github user vikkyrk closed the pull request at:

https://github.com/apache/incubator-beam/pull/1433


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #1433

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 3dbeb8edf -> 1530a1727


Closes #1433


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1530a172
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1530a172
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1530a172

Branch: refs/heads/python-sdk
Commit: 1530a17279d098ae7459f689ef02401f5116e54e
Parents: 3dbeb8e d46203b
Author: Dan Halperin 
Authored: Mon Nov 28 15:54:27 2016 -0800
Committer: Dan Halperin 
Committed: Mon Nov 28 15:54:27 2016 -0800

--
 .../apache_beam/examples/datastore_wordcount.py | 137 +++
 .../apache_beam/io/datastore/v1/datastoreio.py  | 104 +-
 .../io/datastore/v1/datastoreio_test.py |  46 +++
 .../io/datastore/v1/fake_datastore.py   |  17 +++
 .../apache_beam/io/datastore/v1/helper.py   |  35 -
 .../apache_beam/io/datastore/v1/helper_test.py  |  36 +
 .../io/datastore/v1/query_splitter.py   |   7 +-
 7 files changed, 349 insertions(+), 33 deletions(-)
--

[jira] [Commented] (BEAM-1009) Upgrade from mockito-all 1 to mockito-core 2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703565#comment-15703565
 ] 

ASF GitHub Bot commented on BEAM-1009:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1445


> Upgrade from mockito-all 1 to mockito-core 2
> 
>
> Key: BEAM-1009
> URL: https://issues.apache.org/jira/browse/BEAM-1009
> Project: Beam
>  Issue Type: Test
>Reporter: Pei He
>Assignee: Pei He
>
> Mockito 2 provides useful features, and the mockito-all module is no longer 
> generated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: [BEAM-1009] Upgrade byte-buddy version to 1.5.5.

[kenn]

[BEAM-1009] Upgrade byte-buddy version to 1.5.5.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fcd09915
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fcd09915
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fcd09915

Branch: refs/heads/master
Commit: fcd09915fac49f66f6a65f234b34dc34f16fef18
Parents: 0609282
Author: Pei He 
Authored: Mon Nov 28 12:52:14 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 28 15:33:15 2016 -0800

--
 pom.xml |  2 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java| 23 +++-
 .../reflect/ByteBuddyOnTimerInvokerFactory.java | 10 -
 3 files changed, 19 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fcd09915/pom.xml
--
diff --git a/pom.xml b/pom.xml
index e0ef47c..4e60fab 100644
--- a/pom.xml
+++ b/pom.xml
@@ -754,7 +754,7 @@
   
 net.bytebuddy
 byte-buddy
-1.4.3
+1.5.5
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fcd09915/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
index 408ea98..8750d64 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyDoFnInvokerFactory.java
@@ -44,11 +44,11 @@ import net.bytebuddy.implementation.FixedValue;
 import net.bytebuddy.implementation.Implementation;
 import net.bytebuddy.implementation.Implementation.Context;
 import net.bytebuddy.implementation.MethodDelegation;
-import 
net.bytebuddy.implementation.bind.annotation.TargetMethodAnnotationDrivenBinder;
 import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
 import net.bytebuddy.implementation.bytecode.StackManipulation;
 import net.bytebuddy.implementation.bytecode.Throw;
 import net.bytebuddy.implementation.bytecode.assign.Assigner;
+import net.bytebuddy.implementation.bytecode.assign.Assigner.Typing;
 import net.bytebuddy.implementation.bytecode.assign.TypeCasting;
 import net.bytebuddy.implementation.bytecode.constant.TextConstant;
 import net.bytebuddy.implementation.bytecode.member.FieldAccess;
@@ -425,7 +425,7 @@ public class ByteBuddyDoFnInvokerFactory implements 
DoFnInvokerFactory {
   StackManipulation manipulation =
   new StackManipulation.Compound(
   // Push "this" (DoFnInvoker on top of the stack)
-  MethodVariableAccess.REFERENCE.loadOffset(0),
+  MethodVariableAccess.REFERENCE.loadFrom(0),
   // Access this.delegate (DoFn on top of the stack)
   FieldAccess.forField(delegateField).getter(),
   // Cast it to the more precise type
@@ -471,8 +471,11 @@ public class ByteBuddyDoFnInvokerFactory implements 
DoFnInvokerFactory {
  * returns void) or contain the value for the instrumented method to 
return).
  */
 protected StackManipulation afterDelegation(MethodDescription 
instrumentedMethod) {
-  return 
TargetMethodAnnotationDrivenBinder.TerminationHandler.Returning.INSTANCE.resolve(
-  Assigner.DEFAULT, instrumentedMethod, targetMethod);
+  return new StackManipulation.Compound(
+  Assigner.DEFAULT.assign(
+  targetMethod.getReturnType(),
+  instrumentedMethod.getReturnType(), Typing.STATIC),
+  MethodReturn.of(instrumentedMethod.getReturnType()));
 }
   }
 
@@ -491,7 +494,7 @@ public class ByteBuddyDoFnInvokerFactory implements 
DoFnInvokerFactory {
   TypeList.Generic paramTypes = targetMethod.getParameters().asTypeList();
   for (int i = 0; i < paramTypes.size(); i++) {
 TypeDescription.Generic paramT = paramTypes.get(i);
-pushParameters.add(MethodVariableAccess.of(paramT).loadOffset(i + 1));
+pushParameters.add(MethodVariableAccess.of(paramT).loadFrom(i + 1));
 if (!paramT.isPrimitive()) {
   pushParameters.add(TypeCasting.to(paramT));
 }
@@ -634,10 +637,10 @@ public class ByteBuddyDoFnInvokerFactory implements 
DoFnInvokerFactory {
   // To load the delegate, push `this` and then access the field
   StackManipulation pushDelegate =
   new StackManipulation.Compound(
-  

[GitHub] incubator-beam pull request #1445: [BEAM-1009] Upgrade byte-buddy version to...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1445


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: This closes #1445

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 060928257 -> 4bcef03d9


This closes #1445


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4bcef03d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4bcef03d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4bcef03d

Branch: refs/heads/master
Commit: 4bcef03d99e550edf888c1a0b32572f7943cdb2b
Parents: 0609282 fcd0991
Author: Kenneth Knowles 
Authored: Mon Nov 28 15:33:15 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 28 15:33:15 2016 -0800

--
 pom.xml |  2 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java| 23 +++-
 .../reflect/ByteBuddyOnTimerInvokerFactory.java | 10 -
 3 files changed, 19 insertions(+), 16 deletions(-)
--

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_FlinkLocal #939

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1669

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1446: Ignore UsesStatefulParDo Tests in Flink S...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1446


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1446

[kenn]

This closes #1446


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/06092825
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/06092825
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/06092825

Branch: refs/heads/master
Commit: 060928257e809d96fea497c831309040da6fd7c8
Parents: d5aeee9 fff5776
Author: Kenneth Knowles 
Authored: Mon Nov 28 15:13:57 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 28 15:13:57 2016 -0800

--
 runners/flink/runner/pom.xml | 1 +
 1 file changed, 1 insertion(+)
--

[1/2] incubator-beam git commit: Ignore UsesStatefulParDo Tests in Flink Streaming Runner

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master d5aeee9c1 -> 060928257


Ignore UsesStatefulParDo Tests in Flink Streaming Runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fff57761
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fff57761
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fff57761

Branch: refs/heads/master
Commit: fff577610d929e713d42bfb2a4e8428727ee7206
Parents: d5aeee9
Author: Aljoscha Krettek 
Authored: Mon Nov 28 23:33:17 2016 +0100
Committer: Aljoscha Krettek 
Committed: Mon Nov 28 23:33:17 2016 +0100

--
 runners/flink/runner/pom.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fff57761/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index c060c25..18bf6a7 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -79,6 +79,7 @@
 
 
   
org.apache.beam.sdk.testing.RunnableOnService
+  
org.apache.beam.sdk.testing.UsesStatefulParDo
   none
   true
   

[GitHub] incubator-beam pull request #1277: [BEAM-894] Using @Teardown to remove temp...

[peihe]

Github user peihe closed the pull request at:

https://github.com/apache/incubator-beam/pull/1277


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-894) Using @Teardown to remove temp files from failed bundles in Write.WriteBundles

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703431#comment-15703431
 ] 

ASF GitHub Bot commented on BEAM-894:
-

Github user peihe closed the pull request at:

https://github.com/apache/incubator-beam/pull/1277


> Using @Teardown to remove temp files from failed bundles in Write.WriteBundles
> --
>
> Key: BEAM-894
> URL: https://issues.apache.org/jira/browse/BEAM-894
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Pei He
>
> FileBasedSink lefts temp files behind for failed bundles, and it forces 
> finalize() to depend on pattern match.
> However, pattern matching is not always reliable for eventual consistency 
> file system, such as GCS.
> Given we now have DoFn.TearDown, we can improve FileBasedSink (and in general 
> Write transform) to remove temp files/resources early when DoFn bundles fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1036) Support for new State API in FlinkRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703354#comment-15703354
 ] 

ASF GitHub Bot commented on BEAM-1036:
--

GitHub user aljoscha opened a pull request:

https://github.com/apache/incubator-beam/pull/1446

Ignore UsesStatefulParDo Tests in Flink Streaming Runner

Until we fix BEAM-1036 we need to ignore those tests. Before, only the 
tests for the batch runner ignored them.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/incubator-beam 
flink-ignore-state-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1446.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1446


commit fff577610d929e713d42bfb2a4e8428727ee7206
Author: Aljoscha Krettek 
Date:   2016-11-28T22:33:17Z

Ignore UsesStatefulParDo Tests in Flink Streaming Runner




> Support for new State API in FlinkRunner
> 
>
> Key: BEAM-1036
> URL: https://issues.apache.org/jira/browse/BEAM-1036
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Aljoscha Krettek
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1446: Ignore UsesStatefulParDo Tests in Flink S...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/incubator-beam/pull/1446

Ignore UsesStatefulParDo Tests in Flink Streaming Runner

Until we fix BEAM-1036 we need to ignore those tests. Before, only the 
tests for the batch runner ignored them.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/incubator-beam 
flink-ignore-state-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1446.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1446


commit fff577610d929e713d42bfb2a4e8428727ee7206
Author: Aljoscha Krettek 
Date:   2016-11-28T22:33:17Z

Ignore UsesStatefulParDo Tests in Flink Streaming Runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1668

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1009) Upgrade from mockito-all 1 to mockito-core 2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703315#comment-15703315
 ] 

ASF GitHub Bot commented on BEAM-1009:
--

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1445

[BEAM-1009] Upgrade byte-buddy version to 1.5.5.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam mockito-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1445.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1445


commit 1e448e78ee971067606b0b88c71ad1bd00d19cff
Author: Pei He 
Date:   2016-11-28T20:52:14Z

[BEAM-1009] Upgrade byte-buddy version to 1.5.5.




> Upgrade from mockito-all 1 to mockito-core 2
> 
>
> Key: BEAM-1009
> URL: https://issues.apache.org/jira/browse/BEAM-1009
> Project: Beam
>  Issue Type: Test
>Reporter: Pei He
>Assignee: Pei He
>
> Mockito 2 provides useful features, and the mockito-all module is no longer 
> generated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1445: [BEAM-1009] Upgrade byte-buddy version to...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1445

[BEAM-1009] Upgrade byte-buddy version to 1.5.5.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam mockito-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1445.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1445


commit 1e448e78ee971067606b0b88c71ad1bd00d19cff
Author: Pei He 
Date:   2016-11-28T20:52:14Z

[BEAM-1009] Upgrade byte-buddy version to 1.5.5.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1444: [BEAM-1057] Fix JDBC test derby startup i...

[dkulp]

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1444

[BEAM-1057] Fix JDBC test derby startup issues

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam BEAM-1057

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1444.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1444


commit 935553f88f327598320c4f5c1088d490a8a7e8e2
Author: Daniel Kulp 
Date:   2016-11-28T21:47:37Z

[BEAM-1057] Fix JDBC test derby startup issues




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1057) JDBC tests fail to run on slow network

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703243#comment-15703243
 ] 

ASF GitHub Bot commented on BEAM-1057:
--

GitHub user dkulp opened a pull request:

https://github.com/apache/incubator-beam/pull/1444

[BEAM-1057] Fix JDBC test derby startup issues

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dkulp/incubator-beam BEAM-1057

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1444.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1444


commit 935553f88f327598320c4f5c1088d490a8a7e8e2
Author: Daniel Kulp 
Date:   2016-11-28T21:47:37Z

[BEAM-1057] Fix JDBC test derby startup issues




> JDBC tests fail to run on slow network
> --
>
> Key: BEAM-1057
> URL: https://issues.apache.org/jira/browse/BEAM-1057
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Daniel Kulp
>Assignee: Daniel Kulp
>
> In the hotel I'm staying at, the default DNS servers suck which is resulting 
> in the Derby server taking several seconds to startup,  However, the JDBC 
> tests don't "wait" for the server to start.  Thus, the tests all fail.   
> I'll submit a pull request to poll the server for up to 15 seconds to make 
> sure the server starts prior to running the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_RunnableOnService_FlinkLocal #938

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1057) JDBC tests fail to run on slow network

[Daniel Kulp (JIRA)]

Daniel Kulp created BEAM-1057:
-

 Summary: JDBC tests fail to run on slow network
 Key: BEAM-1057
 URL: https://issues.apache.org/jira/browse/BEAM-1057
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Daniel Kulp
Assignee: Daniel Kulp



In the hotel I'm staying at, the default DNS servers suck which is resulting in 
the Derby server taking several seconds to startup,  However, the JDBC tests 
don't "wait" for the server to start.  Thus, the tests all fail.   

I'll submit a pull request to poll the server for up to 15 seconds to make sure 
the server starts prior to running the tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1056) Reference beam version only once in Archetypes

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1056:
-

 Summary: Reference beam version only once in Archetypes
 Key: BEAM-1056
 URL: https://issues.apache.org/jira/browse/BEAM-1056
 Project: Beam
  Issue Type: Improvement
  Components: examples-java
Reporter: Thomas Groh
Priority: Trivial


Instead of referring to the version of the SDK inline, reference it once in a 
beam.version property and use that to determine sdk and runner versions. This 
reduces maintenance burden for updating archetypes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1032) Starter archetype pipeline has no runner on its classpath

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703189#comment-15703189
 ] 

ASF GitHub Bot commented on BEAM-1032:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1414


> Starter archetype pipeline has no runner on its classpath
> -
>
> Key: BEAM-1032
> URL: https://issues.apache.org/jira/browse/BEAM-1032
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> When generating a new maven project using the starter archetype, the pipeline 
> does not execute because it doesn't have a runner jar on it's classpath:
> $ mvn exec:java -Dexec.mainClass=swegner.StarterPipeline
> ...
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java 
> (default-cli) on project yo: An exception occured while executing the Java 
> class. null: InvocationTargetException: No Runner was specified and the 
> DirectRunner was not found on the classpath.
> In the examples archetype, we have the direct runner as a dependency. We 
> should do the same for the starter archetype.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Add direct runner dependency to starter archetype

[tgroh]

Add direct runner dependency to starter archetype


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/14502a31
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/14502a31
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/14502a31

Branch: refs/heads/master
Commit: 14502a3122bd1c1652196881797ffb60d480164e
Parents: 33c6870
Author: Scott Wegner 
Authored: Tue Nov 22 08:56:17 2016 -0800
Committer: Thomas Groh 
Committed: Mon Nov 28 13:17:46 2016 -0800

--
 .../examples/src/main/resources/archetype-resources/pom.xml  | 2 +-
 .../starter/src/main/resources/archetype-resources/pom.xml   | 8 
 .../src/test/resources/projects/basic/reference/pom.xml  | 8 
 3 files changed, 17 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/14502a31/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index b18c57c..031ee88 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -93,7 +93,7 @@
   0.4.0-incubating-SNAPSHOT
 
 
-
+
 
   org.apache.beam
   beam-runners-direct-java

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/14502a31/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
index 738acde..4fae02c 100644
--- 
a/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/main/resources/archetype-resources/pom.xml
@@ -72,6 +72,14 @@
   0.4.0-incubating-SNAPSHOT
 
 
+
+
+  org.apache.beam
+  beam-runners-direct-java
+  0.4.0-incubating-SNAPSHOT
+  runtime
+
+
 
 
   org.slf4j

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/14502a31/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
index 6950ed5..4656e63 100644
--- 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
@@ -72,6 +72,14 @@
   0.4.0-incubating-SNAPSHOT
 
 
+
+
+  org.apache.beam
+  beam-runners-direct-java
+  0.4.0-incubating-SNAPSHOT
+  runtime
+
+
 
 
   org.slf4j

[GitHub] incubator-beam pull request #1414: [BEAM-1032] Add direct runner dependency ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1414


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: This closes #1414

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 33c687069 -> d5aeee9c1


This closes #1414


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d5aeee9c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d5aeee9c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d5aeee9c

Branch: refs/heads/master
Commit: d5aeee9c17a75307ff539e40b3ddc176790ec559
Parents: 33c6870 14502a3
Author: Thomas Groh 
Authored: Mon Nov 28 13:17:46 2016 -0800
Committer: Thomas Groh 
Committed: Mon Nov 28 13:17:46 2016 -0800

--
 .../examples/src/main/resources/archetype-resources/pom.xml  | 2 +-
 .../starter/src/main/resources/archetype-resources/pom.xml   | 8 
 .../src/test/resources/projects/basic/reference/pom.xml  | 8 
 3 files changed, 17 insertions(+), 1 deletion(-)
--

[jira] [Closed] (BEAM-23) User-facing state and timers for ParDo(DoFn)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-23?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles closed BEAM-23.
---
   Resolution: Duplicate
Fix Version/s: Not applicable

> User-facing state and timers for ParDo(DoFn)
> 
>
> Key: BEAM-23
> URL: https://issues.apache.org/jira/browse/BEAM-23
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
> Fix For: Not applicable
>
>
> To a key-partitioned ParDo, add the ability for a user's DoFn to, on a 
> per-key-and-window basis:
>  - read and write consistent state
>  - set a timer to request a callback at a particular moment (in any time 
> domain)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-41) State for "Set" data, supporting efficient membership checks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-41:

Issue Type: New Feature  (was: Sub-task)
Parent: (was: BEAM-23)

> State for "Set" data, supporting efficient membership checks
> 
>
> Key: BEAM-41
> URL: https://issues.apache.org/jira/browse/BEAM-41
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: State
>
> Analogous to how BagState supports efficient blind writes, a SetState would 
> support efficient membership checking without reading the entire set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1399: [BEAM-25] Add support for stateful ParDo ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1399


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-42) State for "Map" data, supporting efficient key lookup, etc.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-42:

Issue Type: New Feature  (was: Sub-task)
Parent: (was: BEAM-23)

> State for "Map" data, supporting efficient key lookup, etc.
> ---
>
> Key: BEAM-42
> URL: https://issues.apache.org/jira/browse/BEAM-42
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: State
>
> Analogous to BagState's blind writes, and SetState's efficient membership 
> check, MapState would add efficient key lookup and mutation, and other 
> Map-related operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703141#comment-15703141
 ] 

ASF GitHub Bot commented on BEAM-25:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1399


> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-27) Add user-ready API for interacting with timers

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-27:

Issue Type: New Feature  (was: Sub-task)
Parent: (was: BEAM-23)

> Add user-ready API for interacting with timers
> --
>
> Key: BEAM-27
> URL: https://issues.apache.org/jira/browse/BEAM-27
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> Pipeline authors will benefit from a different factorization of interaction 
> with underlying timers. The current APIs are targeted at runner implementers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-37) Run DoFnWithContext without conversion to vanilla DoFn

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-37?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-37:

Issue Type: New Feature  (was: Sub-task)
Parent: (was: BEAM-23)

> Run DoFnWithContext without conversion to vanilla DoFn
> --
>
> Key: BEAM-37
> URL: https://issues.apache.org/jira/browse/BEAM-37
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: Not applicable
>
>
> DoFnWithContext is an enhanced DoFn where annotations and parameter lists are 
> inspected to determine whether it accesses windowing information, etc.
> Today, each feature of DoFnWithContext requires implementation on DoFn, which 
> precludes the easy addition of features that we don't have designs for in 
> DoFn.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[5/5] incubator-beam git commit: This closes #1399

[kenn]

This closes #1399


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/33c68706
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/33c68706
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/33c68706

Branch: refs/heads/master
Commit: 33c687069adc1ce4481a0c9637b2a77abb9450b8
Parents: 5e9a80c ec2c0e0
Author: Kenneth Knowles 
Authored: Mon Nov 28 13:03:51 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 28 13:03:51 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java |  60 +++-
 .../beam/runners/direct/EvaluationContext.java  |  15 +
 .../beam/runners/direct/ParDoEvaluator.java |  17 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |  55 +++-
 .../direct/ParDoMultiOverrideFactory.java   |  76 -
 .../ParDoSingleViaMultiOverrideFactory.java |   6 +-
 .../direct/StatefulParDoEvaluatorFactory.java   | 256 
 .../direct/TransformEvaluatorRegistry.java  |   2 +
 .../direct/WatermarkCallbackExecutor.java   |  34 +++
 .../StatefulParDoEvaluatorFactoryTest.java  | 300 +++
 .../org/apache/beam/sdk/transforms/DoFn.java|   4 +-
 .../org/apache/beam/sdk/transforms/OldDoFn.java |   8 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |  10 -
 .../sdk/transforms/reflect/DoFnSignature.java   |   1 +
 .../apache/beam/sdk/transforms/ParDoTest.java   | 177 ++-
 15 files changed, 961 insertions(+), 60 deletions(-)
--

[3/5] incubator-beam git commit: Add State parameter support to SimpleDoFnRunner

[kenn]

Add State parameter support to SimpleDoFnRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e17dc4af
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e17dc4af
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e17dc4af

Branch: refs/heads/master
Commit: e17dc4af9f7de717872d6c6f0ab52e0498f3b782
Parents: 1b7b065
Author: Kenneth Knowles 
Authored: Wed Nov 9 21:10:51 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 28 11:43:21 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java | 60 +--
 .../org/apache/beam/sdk/transforms/ParDo.java   | 10 ---
 .../sdk/transforms/reflect/DoFnSignature.java   |  1 +
 .../apache/beam/sdk/transforms/ParDoTest.java   | 79 
 4 files changed, 118 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e17dc4af/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index f611c0a..68751f0 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -25,7 +25,9 @@ import java.util.Collection;
 import java.util.Iterator;
 import java.util.List;
 import java.util.Set;
+import javax.annotation.Nullable;
 import org.apache.beam.runners.core.DoFnRunners.OutputManager;
+import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.Aggregator.AggregatorFactory;
@@ -37,6 +39,7 @@ import org.apache.beam.sdk.transforms.DoFn.OutputReceiver;
 import org.apache.beam.sdk.transforms.DoFn.ProcessContext;
 import org.apache.beam.sdk.transforms.reflect.DoFnInvoker;
 import org.apache.beam.sdk.transforms.reflect.DoFnInvokers;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
@@ -55,6 +58,10 @@ import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.State;
 import org.apache.beam.sdk.util.state.StateInternals;
+import org.apache.beam.sdk.util.state.StateNamespace;
+import org.apache.beam.sdk.util.state.StateNamespaces;
+import org.apache.beam.sdk.util.state.StateSpec;
+import org.apache.beam.sdk.util.state.StateTags;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
 import org.joda.time.Instant;
@@ -87,6 +94,13 @@ public class SimpleDoFnRunner implements 
DoFnRunner fn,
@@ -98,11 +112,20 @@ public class SimpleDoFnRunner implements 
DoFnRunner(
 options,
@@ -113,7 +136,7 @@ public class SimpleDoFnRunner implements 
DoFnRunner

[4/5] incubator-beam git commit: Add support for Stateful ParDo in the Direct runner

[kenn]

Add support for Stateful ParDo in the Direct runner

This adds overrides and new evaluators to ensure that
state is accessed in a single-threaded manner per key
and is cleaned up when a window expires.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ec2c0e06
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ec2c0e06
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ec2c0e06

Branch: refs/heads/master
Commit: ec2c0e0698c1380b309a609eb642aba445c77e27
Parents: 7e158e4
Author: Kenneth Knowles 
Authored: Wed Nov 9 21:59:15 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 28 11:48:32 2016 -0800

--
 .../beam/runners/direct/EvaluationContext.java  |  15 +
 .../beam/runners/direct/ParDoEvaluator.java |  11 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |  53 +++-
 .../direct/ParDoMultiOverrideFactory.java   |  76 -
 .../ParDoSingleViaMultiOverrideFactory.java |   6 +-
 .../direct/StatefulParDoEvaluatorFactory.java   | 256 
 .../direct/TransformEvaluatorRegistry.java  |   2 +
 .../direct/WatermarkCallbackExecutor.java   |  34 +++
 .../StatefulParDoEvaluatorFactoryTest.java  | 300 +++
 .../org/apache/beam/sdk/transforms/DoFn.java|   4 +-
 .../org/apache/beam/sdk/transforms/OldDoFn.java |   8 +-
 11 files changed, 741 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ec2c0e06/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
index c1225f6..201aaed 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/EvaluationContext.java
@@ -296,6 +296,21 @@ class EvaluationContext {
 fireAvailableCallbacks(lookupProducing(value));
   }
 
+  /**
+   * Schedule a callback to be executed after the given window is expired.
+   *
+   * For example, upstream state associated with the window may be cleared.
+   */
+  public void scheduleAfterWindowExpiration(
+  AppliedPTransform producing,
+  BoundedWindow window,
+  WindowingStrategy windowingStrategy,
+  Runnable runnable) {
+callbackExecutor.callOnWindowExpiration(producing, window, 
windowingStrategy, runnable);
+
+fireAvailableCallbacks(producing);
+  }
+
   private AppliedPTransform getProducing(PValue value) {
 if (value.getProducingTransformInternal() != null) {
   return value.getProducingTransformInternal();

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ec2c0e06/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
index 3285c7e..750e5f1 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
@@ -42,6 +42,7 @@ import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
 
 class ParDoEvaluator implements TransformEvaluator {
+
   public static  ParDoEvaluator create(
   EvaluationContext evaluationContext,
   DirectStepContext stepContext,
@@ -84,11 +85,17 @@ class ParDoEvaluator implements 
TransformEvaluator {
 }
 
 return new ParDoEvaluator<>(
-runner, application, aggregatorChanges, outputBundles.values(), 
stepContext);
+evaluationContext,
+runner,
+application,
+aggregatorChanges,
+outputBundles.values(),
+stepContext);
   }
 
   

 
+  private final EvaluationContext evaluationContext;
   private final PushbackSideInputDoFnRunner fnRunner;
   private final AppliedPTransform transform;
   private final AggregatorContainer.Mutator aggregatorChanges;
@@ -98,11 +105,13 @@ class ParDoEvaluator implements 
TransformEvaluator {
   private final ImmutableList.Builder 
unprocessedElements;
 
   private ParDoEvaluator(
+  EvaluationContext evaluationContext,
   

[1/5] incubator-beam git commit: Remove overspecified type in ParDoEvaluator

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5e9a80cf6 -> 33c687069


Remove overspecified type in ParDoEvaluator


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1b7b065f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1b7b065f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1b7b065f

Branch: refs/heads/master
Commit: 1b7b065f4ccae7c52934b1e73fd1fbfb33c3398d
Parents: 5e9a80c
Author: Kenneth Knowles 
Authored: Mon Nov 21 11:34:45 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 28 11:07:28 2016 -0800

--
 .../java/org/apache/beam/runners/direct/ParDoEvaluator.java| 6 +++---
 .../org/apache/beam/runners/direct/ParDoEvaluatorFactory.java  | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b7b065f/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
index 254fa44..3285c7e 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java
@@ -45,7 +45,7 @@ class ParDoEvaluator implements 
TransformEvaluator {
   public static  ParDoEvaluator create(
   EvaluationContext evaluationContext,
   DirectStepContext stepContext,
-  AppliedPTransform application,
+  AppliedPTransform application,
   WindowingStrategy windowingStrategy,
   Serializable fn, // may be OldDoFn or DoFn
   List sideInputs,
@@ -90,7 +90,7 @@ class ParDoEvaluator implements 
TransformEvaluator {
   

 
   private final PushbackSideInputDoFnRunner fnRunner;
-  private final AppliedPTransform transform;
+  private final AppliedPTransform transform;
   private final AggregatorContainer.Mutator aggregatorChanges;
   private final Collection outputBundles;
   private final DirectStepContext stepContext;
@@ -99,7 +99,7 @@ class ParDoEvaluator implements 
TransformEvaluator {
 
   private ParDoEvaluator(
   PushbackSideInputDoFnRunner fnRunner,
-  AppliedPTransform transform,
+  AppliedPTransform transform,
   AggregatorContainer.Mutator aggregatorChanges,
   Collection outputBundles,
   DirectStepContext stepContext) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1b7b065f/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
index f126000..b776da1 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
@@ -82,7 +82,7 @@ final class ParDoEvaluatorFactory implements 
TransformEvaluator
 try {
   ParDo.BoundMulti transform = application.getTransform();
   return DoFnLifecycleManagerRemovingTransformEvaluator.wrapping(
-  ParDoEvaluator.create(
+  ParDoEvaluator.create(
   evaluationContext,
   stepContext,
   application,

[jira] [Commented] (BEAM-1055) Display Data keys on Python are inconsistent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703118#comment-15703118
 ] 

ASF GitHub Bot commented on BEAM-1055:
--

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1443

[BEAM-1055] Display Data keys on Python are inconsistent

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam beam-1000smth

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1443


commit e620b07afd10439c29401726161cb7139185ec38
Author: Pablo 
Date:   2016-11-28T21:05:47Z

Standardizing display data keys




> Display Data keys on Python are inconsistent
> 
>
> Key: BEAM-1055
> URL: https://issues.apache.org/jira/browse/BEAM-1055
> Project: Beam
>  Issue Type: Bug
>Reporter: Pablo Estrada
>
> Some are in camelCase, some are in snake_case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1443: [BEAM-1055] Display Data keys on Python a...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1443

[BEAM-1055] Display Data keys on Python are inconsistent

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam beam-1000smth

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1443.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1443


commit e620b07afd10439c29401726161cb7139185ec38
Author: Pablo 
Date:   2016-11-28T21:05:47Z

Standardizing display data keys




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-646) Get runners out of the apply()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702974#comment-15702974
 ] 

ASF GitHub Bot commented on BEAM-646:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1442

[BEAM-646] Add Replacement Methods to TransformHierarchy, PValue

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

These are used as the underlying mechanism for Pipeline Surgery.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
transform_hierarchy_surgery_methods

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1442.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1442


commit 09b46520a6a7baa6bd59e69f7bec86a601b5
Author: Thomas Groh 
Date:   2016-11-23T02:19:03Z

Add Replacement Methods to TransformHierarchy, PValue

These are used as the underlying mechanism for Pipeline Surgery.




> Get runners out of the apply()
> --
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now, the runner intercepts calls to apply() and replaces transforms as 
> we go. This means that there is no "original" user graph. For portability and 
> misc architectural benefits, we would like to build the original graph first, 
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more 
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline 
> object.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1442: [BEAM-646] Add Replacement Methods to Tra...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1442

[BEAM-646] Add Replacement Methods to TransformHierarchy, PValue

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

These are used as the underlying mechanism for Pipeline Surgery.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam 
transform_hierarchy_surgery_methods

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1442.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1442


commit 09b46520a6a7baa6bd59e69f7bec86a601b5
Author: Thomas Groh 
Date:   2016-11-23T02:19:03Z

Add Replacement Methods to TransformHierarchy, PValue

These are used as the underlying mechanism for Pipeline Surgery.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1441: Fix double-close bug

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1441

Fix double-close bug

The WritableByteChannel returned for GCS locations has a bug where calling 
close twice throws an Exception, so we cannot safely use AutoCloseable here.

R: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam patch-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1441.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1441


commit 701c3b46c0d1224e513da2278fd4cbe354cdfa5c
Author: sammcveety 
Date:   2016-11-28T19:26:19Z

Fix double-close bug

The WritableByteChannel returned for GCS locations has a bug where calling 
close twice throws an Exception, so we cannot safely use AutoCloseable here.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-991) DatastoreIO Write should flush early for large batches

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702845#comment-15702845
 ] 

Daniel Halperin commented on BEAM-991:
--

Hi [~JoshuaFox], [~vikasrk],

I agree with Vikas, though I would run some performance experiments with 
complex types (proto3 map type with some complex values filled in) to confirm 
there's no perf impact ;)

Joshua, do you know whether you'll be able to take this on?

Thanks,
Dan

> DatastoreIO Write should flush early for large batches
> --
>
> Key: BEAM-991
> URL: https://issues.apache.org/jira/browse/BEAM-991
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>
> If entities are large (avg size > 20KB) then the a single batched write (500 
> entities) would exceed the Datastore size limit of a single request (10MB) 
> from https://cloud.google.com/datastore/docs/concepts/limits.
> First reported in: 
> http://stackoverflow.com/questions/40156400/why-does-dataflow-erratically-fail-in-datastore-access



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1055) Display Data keys on Python are inconsistent

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-1055:
---

 Summary: Display Data keys on Python are inconsistent
 Key: BEAM-1055
 URL: https://issues.apache.org/jira/browse/BEAM-1055
 Project: Beam
  Issue Type: Bug
Reporter: Pablo Estrada


Some are in camelCase, some are in snake_case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1054) Language specific IO docs

[Melissa Pashniak (JIRA)]

Melissa Pashniak created BEAM-1054:
--

 Summary: Language specific IO docs
 Key: BEAM-1054
 URL: https://issues.apache.org/jira/browse/BEAM-1054
 Project: Beam
  Issue Type: Task
  Components: website
Reporter: Melissa Pashniak
Assignee: Melissa Pashniak






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1006) splitKeysToFilter method is creating a bad query string.

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702741#comment-15702741
 ] 

Jean-Baptiste Onofré commented on BEAM-1006:


Sorry, I've missing this Jira. The PR looks good. I'm building and merging.

> splitKeysToFilter method is creating a bad query string.
> 
>
> Key: BEAM-1006
> URL: https://issues.apache.org/jira/browse/BEAM-1006
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.3.0-incubating
>Reporter: David Billings
>Assignee: Jean-Baptiste Onofré
>Priority: Critical
>  Labels: mongodb
> Fix For: 0.4.0-incubating
>
>
> When you have a large number of documents in your mongo collection the 
> splitKeysToFilter method is invoked. 
> Currently Document{{_id= is prefixed to the query string
> {code}{ $and: [ 
> {"_id":{$lte:Objectd("Document{{_id=564a0ad6e21b840001d75f5d}}")}} ]}{code}
> This causes the document parse to fail with the following exception: 
> {code}Exception in thread "main" java.lang.IllegalArgumentException: invalid 
> hexadecimal representation of an ObjectId: 
> [Document{{_id=56e1ec546b2f0700015e61b2}}]{code}
> This is caused by line 305 
> {code}
> String splitKey = splitKeys.get(i).toString();
> {code}
> The code should use 
> {code}
> String splitKey = splitKeys.get(i).get("_id").toString();
> {code}
> to just return the ID. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-795) org.apache.beam.sdk.io.ReadTest.testStreamingPrimitiveDisplayData failing in Dataflow RunnableOnService

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-795.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> org.apache.beam.sdk.io.ReadTest.testStreamingPrimitiveDisplayData failing in 
> Dataflow RunnableOnService
> ---
>
> Key: BEAM-795
> URL: https://issues.apache.org/jira/browse/BEAM-795
> Project: Beam
>  Issue Type: Bug
>Reporter: Jason Kuster
>Assignee: Scott Wegner
>Priority: Critical
> Fix For: Not applicable
>
>
> Test is failing in beam_PostCommit_RunnableOnService_GoogleCloudDataflow 
> since yesterday. Culprit change seems to be 
> https://github.com/apache/incubator-beam/commit/ad03d07ae783f054a31e8b2e14100afff8cdf747
> Affected Jenkins runs:
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/
> 1381 - 1386, 1388 as of bug creation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-230) Remove WindowedValue#valueInEmptyWindows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702717#comment-15702717
 ] 

ASF GitHub Bot commented on BEAM-230:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1440

[BEAM-230] Remove WindowedValue.valueInEmptyWindows

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

A value in empty windows expands to no values, so it can be dropped at any 
time, perhaps unintentionally. This has affected real runners.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam valueInEmptyWindows

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1440.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1440


commit 05fb97b7133d79f343d0d6d6b26a3a56f8ee848b
Author: Kenneth Knowles 
Date:   2016-05-10T18:39:35Z

Remove WindowedValue.valueInEmptyWindows

A value in empty windows expands to no values, so it can be dropped at
any time, perhaps unintentionally. This has bitten runner authors, including
Spark & Dataflow.

While creating such a thing in memory is not automatically problematic, it
is also not really useful. So this change removes it.




> Remove WindowedValue#valueInEmptyWindows
> 
>
> Key: BEAM-230
> URL: https://issues.apache.org/jira/browse/BEAM-230
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: Not applicable
>
>
> A WindowedValue in no windows does not exist, and can be dropped by a runner 
> at any time.
> We should also assert that any collection of windows is nonempty when 
> creating a new WindowedValue. If a user wants to drop an element, they should 
> explicitly filter it out rather than expecting it to be dropped by the runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1440: [BEAM-230] Remove WindowedValue.valueInEm...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1440

[BEAM-230] Remove WindowedValue.valueInEmptyWindows

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

A value in empty windows expands to no values, so it can be dropped at any 
time, perhaps unintentionally. This has affected real runners.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam valueInEmptyWindows

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1440.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1440


commit 05fb97b7133d79f343d0d6d6b26a3a56f8ee848b
Author: Kenneth Knowles 
Date:   2016-05-10T18:39:35Z

Remove WindowedValue.valueInEmptyWindows

A value in empty windows expands to no values, so it can be dropped at
any time, perhaps unintentionally. This has bitten runner authors, including
Spark & Dataflow.

While creating such a thing in memory is not automatically problematic, it
is also not really useful. So this change removes it.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Reopened] (BEAM-230) Remove WindowedValue#valueInEmptyWindows

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-230:
--
  Assignee: Kenneth Knowles  (was: Thomas Groh)

> Remove WindowedValue#valueInEmptyWindows
> 
>
> Key: BEAM-230
> URL: https://issues.apache.org/jira/browse/BEAM-230
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: Not applicable
>
>
> A WindowedValue in no windows does not exist, and can be dropped by a runner 
> at any time.
> We should also assert that any collection of windows is nonempty when 
> creating a new WindowedValue. If a user wants to drop an element, they should 
> explicitly filter it out rather than expecting it to be dropped by the runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-230) Remove WindowedValue#valueInEmptyWindows

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702714#comment-15702714
 ] 

Kenneth Knowles commented on BEAM-230:
--

This wasn't actually removed completely.

> Remove WindowedValue#valueInEmptyWindows
> 
>
> Key: BEAM-230
> URL: https://issues.apache.org/jira/browse/BEAM-230
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: Not applicable
>
>
> A WindowedValue in no windows does not exist, and can be dropped by a runner 
> at any time.
> We should also assert that any collection of windows is nonempty when 
> creating a new WindowedValue. If a user wants to drop an element, they should 
> explicitly filter it out rather than expecting it to be dropped by the runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1424: Add input type to TransformResult

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1424


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Add input type to TransformResult

[tgroh]

Add input type to TransformResult

This would likely have caught some hard-to-diagnose type safety errors
during the development of StatefulParDoEvaluatorFactory, so adding it
should hopefully catch similar bugs in the future.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7502adda
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7502adda
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7502adda

Branch: refs/heads/master
Commit: 7502adda3262bce9d6d4fe4499bde8d8b5273029
Parents: 9fbd2d2
Author: Kenneth Knowles 
Authored: Tue Nov 22 16:01:45 2016 -0800
Committer: Thomas Groh 
Committed: Mon Nov 28 10:06:31 2016 -0800

--
 .../direct/AbstractModelEnforcement.java|  2 +-
 .../direct/BoundedReadEvaluatorFactory.java |  2 +-
 .../beam/runners/direct/CommittedResult.java|  2 +-
 .../beam/runners/direct/CompletionCallback.java |  2 +-
 ...ecycleManagerRemovingTransformEvaluator.java |  2 +-
 .../runners/direct/EmptyTransformEvaluator.java |  4 +-
 .../beam/runners/direct/EvaluationContext.java  |  2 +-
 .../direct/ExecutorServiceParallelExecutor.java |  2 +-
 .../runners/direct/FlattenEvaluatorFactory.java | 10 ++---
 .../GroupAlsoByWindowEvaluatorFactory.java  |  5 ++-
 .../direct/GroupByKeyOnlyEvaluatorFactory.java  |  2 +-
 .../direct/ImmutabilityEnforcementFactory.java  |  2 +-
 .../beam/runners/direct/ModelEnforcement.java   |  2 +-
 .../beam/runners/direct/ParDoEvaluator.java |  2 +-
 .../direct/PassthroughTransformEvaluator.java   |  4 +-
 .../runners/direct/StepTransformResult.java | 38 +
 .../direct/TestStreamEvaluatorFactory.java  |  2 +-
 .../beam/runners/direct/TransformEvaluator.java |  2 +-
 .../beam/runners/direct/TransformExecutor.java  |  4 +-
 .../beam/runners/direct/TransformResult.java| 16 +--
 .../direct/UnboundedReadEvaluatorFactory.java   |  3 +-
 .../runners/direct/ViewEvaluatorFactory.java|  2 +-
 .../runners/direct/WindowEvaluatorFactory.java  |  6 ++-
 .../direct/BoundedReadEvaluatorFactoryTest.java | 10 ++---
 ...leManagerRemovingTransformEvaluatorTest.java |  4 +-
 .../runners/direct/EvaluationContextTest.java   | 20 -
 .../direct/FlattenEvaluatorFactoryTest.java |  6 +--
 .../ImmutabilityEnforcementFactoryTest.java |  6 +--
 .../beam/runners/direct/ParDoEvaluatorTest.java |  2 +-
 .../runners/direct/StepTransformResultTest.java | 25 ++-
 .../direct/TestStreamEvaluatorFactoryTest.java  | 10 ++---
 .../runners/direct/TransformExecutorTest.java   | 45 ++--
 .../UnboundedReadEvaluatorFactoryTest.java  | 20 ++---
 .../direct/WindowEvaluatorFactoryTest.java  | 12 +++---
 34 files changed, 152 insertions(+), 126 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7502adda/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AbstractModelEnforcement.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AbstractModelEnforcement.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AbstractModelEnforcement.java
index 81f0f5f..f09164b 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AbstractModelEnforcement.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/AbstractModelEnforcement.java
@@ -33,6 +33,6 @@ abstract class AbstractModelEnforcement implements 
ModelEnforcement {
   @Override
   public void afterFinish(
   CommittedBundle input,
-  TransformResult result,
+  TransformResult result,
   Iterable> outputs) {}
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7502adda/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
index 66c55cd..65b622f 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
@@ -161,7 +161,7 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 }
 
 @Override
-public TransformResult finishBundle()  {
+public TransformResult finishBundle()  {
   return resultBuilder.build();
 }
   }

[1/2] incubator-beam git commit: This closes #1424

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 9fbd2d24e -> 5e9a80cf6


This closes #1424


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5e9a80cf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5e9a80cf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5e9a80cf

Branch: refs/heads/master
Commit: 5e9a80cf61c4666edb5febd9a0fface2d2c26261
Parents: 9fbd2d2 7502add
Author: Thomas Groh 
Authored: Mon Nov 28 10:06:31 2016 -0800
Committer: Thomas Groh 
Committed: Mon Nov 28 10:06:31 2016 -0800

--
 .../direct/AbstractModelEnforcement.java|  2 +-
 .../direct/BoundedReadEvaluatorFactory.java |  2 +-
 .../beam/runners/direct/CommittedResult.java|  2 +-
 .../beam/runners/direct/CompletionCallback.java |  2 +-
 ...ecycleManagerRemovingTransformEvaluator.java |  2 +-
 .../runners/direct/EmptyTransformEvaluator.java |  4 +-
 .../beam/runners/direct/EvaluationContext.java  |  2 +-
 .../direct/ExecutorServiceParallelExecutor.java |  2 +-
 .../runners/direct/FlattenEvaluatorFactory.java | 10 ++---
 .../GroupAlsoByWindowEvaluatorFactory.java  |  5 ++-
 .../direct/GroupByKeyOnlyEvaluatorFactory.java  |  2 +-
 .../direct/ImmutabilityEnforcementFactory.java  |  2 +-
 .../beam/runners/direct/ModelEnforcement.java   |  2 +-
 .../beam/runners/direct/ParDoEvaluator.java |  2 +-
 .../direct/PassthroughTransformEvaluator.java   |  4 +-
 .../runners/direct/StepTransformResult.java | 38 +
 .../direct/TestStreamEvaluatorFactory.java  |  2 +-
 .../beam/runners/direct/TransformEvaluator.java |  2 +-
 .../beam/runners/direct/TransformExecutor.java  |  4 +-
 .../beam/runners/direct/TransformResult.java| 16 +--
 .../direct/UnboundedReadEvaluatorFactory.java   |  3 +-
 .../runners/direct/ViewEvaluatorFactory.java|  2 +-
 .../runners/direct/WindowEvaluatorFactory.java  |  6 ++-
 .../direct/BoundedReadEvaluatorFactoryTest.java | 10 ++---
 ...leManagerRemovingTransformEvaluatorTest.java |  4 +-
 .../runners/direct/EvaluationContextTest.java   | 20 -
 .../direct/FlattenEvaluatorFactoryTest.java |  6 +--
 .../ImmutabilityEnforcementFactoryTest.java |  6 +--
 .../beam/runners/direct/ParDoEvaluatorTest.java |  2 +-
 .../runners/direct/StepTransformResultTest.java | 25 ++-
 .../direct/TestStreamEvaluatorFactoryTest.java  | 10 ++---
 .../runners/direct/TransformExecutorTest.java   | 45 ++--
 .../UnboundedReadEvaluatorFactoryTest.java  | 20 ++---
 .../direct/WindowEvaluatorFactoryTest.java  | 12 +++---
 34 files changed, 152 insertions(+), 126 deletions(-)
--

[jira] [Commented] (BEAM-975) Issue with MongoDBIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702604#comment-15702604
 ] 

Jean-Baptiste Onofré commented on BEAM-975:
---

I would like to include this for 0.4.0-incubating release.

> Issue with MongoDBIO
> 
>
> Key: BEAM-975
> URL: https://issues.apache.org/jira/browse/BEAM-975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Reza Nouri
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.4.0-incubating
>
>
> It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
> REST service that reads data from Mongo. After a number of requests, mongoIO 
> throws the following exception:
> com.mongodb.MongoSocketReadException: Prematurely reached end of stream
>   at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
>   at 
> com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
>   at 
> com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
>   at 
> com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
>   at 
> com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
>   at 
> com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
>   at 
> com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
>   at 
> com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
>   at 
> com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
>   at 
> com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
>   at 
> com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
>   at com.mongodb.Mongo.execute(Mongo.java:772)
>   at com.mongodb.Mongo$2.execute(Mongo.java:759)
>   at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
>   at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I suppose there must be a problem with Mongo connection which causes this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)