[jira] [Commented] (BEAM-1061) PreCommit test with side inputs

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707749#comment-15707749
 ] 

Daniel Halperin commented on BEAM-1061:
---

It is covered by ROS tests, but they do not run precommit.

> PreCommit test with side inputs
> ---
>
> Key: BEAM-1061
> URL: https://issues.apache.org/jira/browse/BEAM-1061
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> We should have at least one precommit integration test that exercises side 
> inputs on all runners. Existing tests exercise sources, files, per-key, 
> combiners, windowing, ...; side inputs is one lacking part of the model it 
> would be nice to touch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1062) Shade SDK based on a whitelist instead of a blacklist

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707464#comment-15707464
 ] 

ASF GitHub Bot commented on BEAM-1062:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1462

[BEAM-1062] Shade the SDK more fully; everything but a whitelist

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This is a re-opening of a past change. Previously, issues with snappy's use 
of reflection blocked progress. Things have changed a lot since then so let's 
put a little more effort into this.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam shade-SDK

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1462.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1462


commit 78dd9e3583349d232c2514a0380fa59b0bbd2791
Author: Kenneth Knowles 
Date:   2016-07-15T20:47:41Z

Upgrade snappy version and move to dependencyManagement

commit 05214785fa254ee434e8fdb0ae728c8505cb1cdd
Author: Kenneth Knowles 
Date:   2016-11-30T04:06:54Z

Shade more in the SDK; everything except a whitelist




> Shade SDK based on a whitelist instead of a blacklist
> -
>
> Key: BEAM-1062
> URL: https://issues.apache.org/jira/browse/BEAM-1062
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> This is a more robust way to manage the surface of dependencies we introduce.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1462: [BEAM-1062] Shade the SDK more fully; eve...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1462

[BEAM-1062] Shade the SDK more fully; everything but a whitelist

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This is a re-opening of a past change. Previously, issues with snappy's use 
of reflection blocked progress. Things have changed a lot since then so let's 
put a little more effort into this.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam shade-SDK

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1462.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1462


commit 78dd9e3583349d232c2514a0380fa59b0bbd2791
Author: Kenneth Knowles 
Date:   2016-07-15T20:47:41Z

Upgrade snappy version and move to dependencyManagement

commit 05214785fa254ee434e8fdb0ae728c8505cb1cdd
Author: Kenneth Knowles 
Date:   2016-11-30T04:06:54Z

Shade more in the SDK; everything except a whitelist




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1062) Shade SDK based on a whitelist instead of a blacklist

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1062:
-

 Summary: Shade SDK based on a whitelist instead of a blacklist
 Key: BEAM-1062
 URL: https://issues.apache.org/jira/browse/BEAM-1062
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


This is a more robust way to manage the surface of dependencies we introduce.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1685

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1061) PreCommit test with side inputs

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707206#comment-15707206
 ] 

Luke Cwik commented on BEAM-1061:
-

Is this not covered by the RunnableOnService tests found in ViewTest.java?

> PreCommit test with side inputs
> ---
>
> Key: BEAM-1061
> URL: https://issues.apache.org/jira/browse/BEAM-1061
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> We should have at least one precommit integration test that exercises side 
> inputs on all runners. Existing tests exercise sources, files, per-key, 
> combiners, windowing, ...; side inputs is one lacking part of the model it 
> would be nice to touch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1061) PreCommit test with side inputs

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1061:
-

 Summary: PreCommit test with side inputs
 Key: BEAM-1061
 URL: https://issues.apache.org/jira/browse/BEAM-1061
 Project: Beam
  Issue Type: Test
  Components: testing
Reporter: Daniel Halperin
Assignee: Daniel Halperin


We should have at least one precommit integration test that exercises side 
inputs on all runners. Existing tests exercise sources, files, per-key, 
combiners, windowing, ...; side inputs is one lacking part of the model it 
would be nice to touch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #1460

[dhalperi]

Closes #1460


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8042d52f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8042d52f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8042d52f

Branch: refs/heads/master
Commit: 8042d52fcb377922a11b9cc5f548690da83a2b1c
Parents: b1f7013 98ab559
Author: Dan Halperin 
Authored: Tue Nov 29 17:40:51 2016 -0800
Committer: Dan Halperin 
Committed: Tue Nov 29 17:40:51 2016 -0800

--
 .../direct/FlattenEvaluatorFactoryTest.java |  8 ++---
 .../beam/runners/dataflow/DataflowRunner.java   | 10 +++---
 .../org/apache/beam/sdk/util/WindowedValue.java | 33 +---
 .../beam/sdk/testing/PaneExtractorsTest.java|  2 +-
 .../apache/beam/sdk/util/WindowedValueTest.java | 10 ++
 5 files changed, 49 insertions(+), 14 deletions(-)
--

[1/2] incubator-beam git commit: Revert "Remove WindowedValue.valueInEmptyWindows"

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master b1f7013d8 -> 8042d52fc


Revert "Remove WindowedValue.valueInEmptyWindows"

This reverts commit 0e49b150e83d85ae432c640da937a9497068e71b, which breaks
some DataflowRunner integration tests.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/98ab5594
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/98ab5594
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/98ab5594

Branch: refs/heads/master
Commit: 98ab559410bde425c9c1944bcd2f09293c3764dc
Parents: b1f7013
Author: Kenneth Knowles 
Authored: Tue Nov 29 16:57:09 2016 -0800
Committer: Dan Halperin 
Committed: Tue Nov 29 17:40:50 2016 -0800

--
 .../direct/FlattenEvaluatorFactoryTest.java |  8 ++---
 .../beam/runners/dataflow/DataflowRunner.java   | 10 +++---
 .../org/apache/beam/sdk/util/WindowedValue.java | 33 +---
 .../beam/sdk/testing/PaneExtractorsTest.java|  2 +-
 .../apache/beam/sdk/util/WindowedValueTest.java | 10 ++
 5 files changed, 49 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/98ab5594/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
index 39c7cab..cb27fbc 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
@@ -78,9 +78,9 @@ public class FlattenEvaluatorFactoryTest {
 rightSideEvaluator.processElement(WindowedValue.valueInGlobalWindow(-1));
 leftSideEvaluator.processElement(
 WindowedValue.timestampedValueInGlobalWindow(2, new Instant(1024)));
-leftSideEvaluator.processElement(WindowedValue.valueInGlobalWindow(4, 
PaneInfo.NO_FIRING));
+leftSideEvaluator.processElement(WindowedValue.valueInEmptyWindows(4, 
PaneInfo.NO_FIRING));
 rightSideEvaluator.processElement(
-WindowedValue.valueInGlobalWindow(2, 
PaneInfo.ON_TIME_AND_ONLY_FIRING));
+WindowedValue.valueInEmptyWindows(2, 
PaneInfo.ON_TIME_AND_ONLY_FIRING));
 rightSideEvaluator.processElement(
 WindowedValue.timestampedValueInGlobalWindow(-4, new Instant(-4096)));
 
@@ -104,12 +104,12 @@ public class FlattenEvaluatorFactoryTest {
 flattenedLeftBundle.commit(Instant.now()).getElements(),
 containsInAnyOrder(
 WindowedValue.timestampedValueInGlobalWindow(2, new Instant(1024)),
-WindowedValue.valueInGlobalWindow(4, PaneInfo.NO_FIRING),
+WindowedValue.valueInEmptyWindows(4, PaneInfo.NO_FIRING),
 WindowedValue.valueInGlobalWindow(1)));
 assertThat(
 flattenedRightBundle.commit(Instant.now()).getElements(),
 containsInAnyOrder(
-WindowedValue.valueInGlobalWindow(2, 
PaneInfo.ON_TIME_AND_ONLY_FIRING),
+WindowedValue.valueInEmptyWindows(2, 
PaneInfo.ON_TIME_AND_ONLY_FIRING),
 WindowedValue.timestampedValueInGlobalWindow(-4, new 
Instant(-4096)),
 WindowedValue.valueInGlobalWindow(-1)));
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/98ab5594/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 641daf4..0099856 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -23,7 +23,7 @@ import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.base.Strings.isNullOrEmpty;
 import static org.apache.beam.sdk.util.StringUtils.approximatePTransformName;
 import static org.apache.beam.sdk.util.StringUtils.approximateSimpleName;
-import static org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
+import static org.apache.beam.sdk.util.WindowedValue.valueInEmptyWindows;
 
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;
@@ -1230,7 +1230,7 @@ public class DataflowRunner extends 
PipelineRunner {
 // are at a windo

[jira] [Created] (BEAM-1060) Make DoFnTester use new DoFn

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1060:
--

 Summary: Make DoFnTester use new DoFn
 Key: BEAM-1060
 URL: https://issues.apache.org/jira/browse/BEAM-1060
 Project: Beam
  Issue Type: Improvement
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1060) Make DoFnTester use new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707197#comment-15707197
 ] 

ASF GitHub Bot commented on BEAM-1060:
--

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1461

[BEAM-1060] Makes DoFnTester use new DoFn internally.

There were 2 remaining users of DoFnTester.of(OldDoFn):
- SplittableParDo.ProcessElements: this is fixed in
  https://github.com/apache/incubator-beam/pull/1261
- GroupAlsoByWindowsProperties: this one is harder.
  Various GABWDoFn's use OldDoFn.windowingInternals,
  and we can't pass that through a new DoFn.
  So instead I removed usage of DoFnTester from
  GroupAlsoByWindowsProperties in favor of a tiny
  hand-coded solution.

So after this PR and #1261 DoFnTester.of(OldDoFn) can be deleted.

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam tester-new-dofn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1461.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1461


commit 4cdccc044f6d7b7599860c3bc656a7b33905f79f
Author: Eugene Kirpichov 
Date:   2016-11-18T21:10:22Z

Makes DoFnTester use new DoFn internally.

There were 2 remaining users of DoFnTester.of(OldDoFn):
- SplittableParDo.ProcessElements: this is fixed in
  https://github.com/apache/incubator-beam/pull/1261
- GroupAlsoByWindowsProperties: this one is harder.
  Various GABWDoFn's use OldDoFn.windowingInternals,
  and we can't pass that through a new DoFn.
  So instead I removed usage of DoFnTester from
  GroupAlsoByWindowsProperties in favor of a tiny
  hand-coded solution.

So after #1261 DoFnTester.of(OldDoFn) can be deleted.




> Make DoFnTester use new DoFn
> 
>
> Key: BEAM-1060
> URL: https://issues.apache.org/jira/browse/BEAM-1060
> Project: Beam
>  Issue Type: Improvement
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1460: Revert "Remove WindowedValue.valueInEmpty...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1460


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1461: [BEAM-1060] Makes DoFnTester use new DoFn...

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1461

[BEAM-1060] Makes DoFnTester use new DoFn internally.

There were 2 remaining users of DoFnTester.of(OldDoFn):
- SplittableParDo.ProcessElements: this is fixed in
  https://github.com/apache/incubator-beam/pull/1261
- GroupAlsoByWindowsProperties: this one is harder.
  Various GABWDoFn's use OldDoFn.windowingInternals,
  and we can't pass that through a new DoFn.
  So instead I removed usage of DoFnTester from
  GroupAlsoByWindowsProperties in favor of a tiny
  hand-coded solution.

So after this PR and #1261 DoFnTester.of(OldDoFn) can be deleted.

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam tester-new-dofn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1461.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1461


commit 4cdccc044f6d7b7599860c3bc656a7b33905f79f
Author: Eugene Kirpichov 
Date:   2016-11-18T21:10:22Z

Makes DoFnTester use new DoFn internally.

There were 2 remaining users of DoFnTester.of(OldDoFn):
- SplittableParDo.ProcessElements: this is fixed in
  https://github.com/apache/incubator-beam/pull/1261
- GroupAlsoByWindowsProperties: this one is harder.
  Various GABWDoFn's use OldDoFn.windowingInternals,
  and we can't pass that through a new DoFn.
  So instead I removed usage of DoFnTester from
  GroupAlsoByWindowsProperties in favor of a tiny
  hand-coded solution.

So after #1261 DoFnTester.of(OldDoFn) can be deleted.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1684

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1458: Restore a private ValueInEmptyWindows to ...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1458


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707098#comment-15707098
 ] 

ASF GitHub Bot commented on BEAM-551:
-

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1459

[BEAM-551] Changes to BigQuery validation to support dynamic values

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/bqstream

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1459.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1459


commit e8d5e82748bfa1c2801cd06291c73f7d12799cc8
Author: Sam McVeety 
Date:   2016-11-30T00:55:48Z

Fix tests




> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1460: Revert "Remove WindowedValue.valueInEmpty...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1460

Revert "Remove WindowedValue.valueInEmptyWindows"

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This reverts commit 0e49b150e83d85ae432c640da937a9497068e71b, which breaks 
some DataflowRunner integration tests in ways that may take a minute to fix.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam rollback

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1460.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1460


commit 54435de54eb7d4bff8db2ae944bea1f8e813ecbd
Author: Kenneth Knowles 
Date:   2016-11-30T00:57:09Z

Revert "Remove WindowedValue.valueInEmptyWindows"

This reverts commit 0e49b150e83d85ae432c640da937a9497068e71b, which breaks
some DataflowRunner integration tests.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1459: [BEAM-551] Changes to BigQuery validation...

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1459

[BEAM-551] Changes to BigQuery validation to support dynamic values

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/bqstream

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1459.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1459


commit e8d5e82748bfa1c2801cd06291c73f7d12799cc8
Author: Sam McVeety 
Date:   2016-11-30T00:55:48Z

Fix tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1683

[Apache Jenkins Server]

See 

[1/2] incubator-beam git commit: This closes #1382

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 5ce75a2ea -> 70c1de9b9


This closes #1382


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/70c1de9b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/70c1de9b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/70c1de9b

Branch: refs/heads/python-sdk
Commit: 70c1de9b95e9c20e5efb277d9ad50ae6348418e0
Parents: 5ce75a2 81e7a0f
Author: Thomas Groh 
Authored: Tue Nov 29 15:43:04 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 29 15:43:04 2016 -0800

--
 sdks/python/apache_beam/dataflow_test.py | 66 +++
 sdks/python/apache_beam/test_pipeline.py | 76 +++
 2 files changed, 120 insertions(+), 22 deletions(-)
--

[2/2] incubator-beam git commit: Support ValidatesRunner Attribute in Python

[tgroh]

Support ValidatesRunner Attribute in Python

This is roughly equivalent to "RunnableOnService" in the Java SDK. See
BEAM-655


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/81e7a0f6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/81e7a0f6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/81e7a0f6

Branch: refs/heads/python-sdk
Commit: 81e7a0f653864212a5c9d3d0802608f92bb34501
Parents: 5ce75a2
Author: Mark Liu 
Authored: Thu Nov 17 14:45:42 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 29 15:43:04 2016 -0800

--
 sdks/python/apache_beam/dataflow_test.py | 66 +++
 sdks/python/apache_beam/test_pipeline.py | 76 +++
 2 files changed, 120 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/81e7a0f6/sdks/python/apache_beam/dataflow_test.py
--
diff --git a/sdks/python/apache_beam/dataflow_test.py 
b/sdks/python/apache_beam/dataflow_test.py
index f96e8af..ba3553a 100644
--- a/sdks/python/apache_beam/dataflow_test.py
+++ b/sdks/python/apache_beam/dataflow_test.py
@@ -24,13 +24,13 @@ import re
 import unittest
 
 import apache_beam as beam
-from apache_beam.pipeline import Pipeline
 from apache_beam.pvalue import AsDict
 from apache_beam.pvalue import AsIter as AllOf
 from apache_beam.pvalue import AsList
 from apache_beam.pvalue import AsSingleton
 from apache_beam.pvalue import EmptySideInput
 from apache_beam.pvalue import SideOutputValue
+from apache_beam.test_pipeline import TestPipeline
 from apache_beam.transforms import Create
 from apache_beam.transforms import DoFn
 from apache_beam.transforms import FlatMap
@@ -42,6 +42,7 @@ from apache_beam.transforms.util import assert_that
 from apache_beam.transforms.util import equal_to
 from apache_beam.transforms.window import IntervalWindow
 from apache_beam.transforms.window import WindowFn
+from nose.plugins.attrib import attr
 
 
 class DataflowTest(unittest.TestCase):
@@ -58,8 +59,9 @@ class DataflowTest(unittest.TestCase):
 | 'GroupCounts' >> GroupByKey()
 | 'AddCounts' >> Map(lambda (x, ones): (x, sum(ones
 
+  @attr('ValidatesRunner')
   def test_word_count(self):
-pipeline = Pipeline('DirectPipelineRunner')
+pipeline = TestPipeline()
 lines = pipeline | 'SomeWords' >> Create(DataflowTest.SAMPLE_DATA)
 result = (
 (lines | 'GetWords' >> FlatMap(lambda x: re.findall(r'\w+', x)))
@@ -67,8 +69,9 @@ class DataflowTest(unittest.TestCase):
 assert_that(result, equal_to(DataflowTest.SAMPLE_RESULT))
 pipeline.run()
 
+  @attr('ValidatesRunner')
   def test_map(self):
-pipeline = Pipeline('DirectPipelineRunner')
+pipeline = TestPipeline()
 lines = pipeline | 'input' >> Create(['a', 'b', 'c'])
 result = (lines
   | 'upper' >> Map(str.upper)
@@ -76,8 +79,9 @@ class DataflowTest(unittest.TestCase):
 assert_that(result, equal_to(['foo-A', 'foo-B', 'foo-C']))
 pipeline.run()
 
+  @attr('ValidatesRunner')
   def test_par_do_with_side_input_as_arg(self):
-pipeline = Pipeline('DirectPipelineRunner')
+pipeline = TestPipeline()
 words_list = ['aa', 'bb', 'cc']
 words = pipeline | 'SomeWords' >> Create(words_list)
 prefix = pipeline | 'SomeString' >> Create(['xyz'])  # side in
@@ -89,8 +93,9 @@ class DataflowTest(unittest.TestCase):
 assert_that(result, equal_to(['xyz-%s-zyx' % x for x in words_list]))
 pipeline.run()
 
+  @attr('ValidatesRunner')
   def test_par_do_with_side_input_as_keyword_arg(self):
-pipeline = Pipeline('DirectPipelineRunner')
+pipeline = TestPipeline()
 words_list = ['aa', 'bb', 'cc']
 words = pipeline | 'SomeWords' >> Create(words_list)
 prefix = 'zyx'
@@ -102,6 +107,7 @@ class DataflowTest(unittest.TestCase):
 assert_that(result, equal_to(['zyx-%s-xyz' % x for x in words_list]))
 pipeline.run()
 
+  @attr('ValidatesRunner')
   def test_par_do_with_do_fn_object(self):
 class SomeDoFn(DoFn):
   """A custom DoFn for a FlatMap transform."""
@@ -109,7 +115,7 @@ class DataflowTest(unittest.TestCase):
   def process(self, context, prefix, suffix):
 return ['%s-%s-%s' % (prefix, context.element, suffix)]
 
-pipeline = Pipeline('DirectPipelineRunner')
+pipeline = TestPipeline()
 words_list = ['aa', 'bb', 'cc']
 words = pipeline | 'SomeWords' >> Create(words_list)
 prefix = 'zyx'
@@ -119,6 +125,7 @@ class DataflowTest(unittest.TestCase):
 assert_that(result, equal_to(['zyx-%s-xyz' % x for x in words_list]))
 pipeline.run()
 
+  @attr('ValidatesRunner')
   def test_par_do_with_multiple_outputs_and_using_yield(self):
 class SomeDoFn(DoFn):
   """A c

[GitHub] incubator-beam pull request #1457: Remove TransformApplicationsForTesting

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1457


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1457

[tgroh]

This closes #1457


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b1f7013d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b1f7013d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b1f7013d

Branch: refs/heads/master
Commit: b1f7013d88cea1290f92898b36d6a761546f8e60
Parents: 8d127be 1184bfa
Author: Thomas Groh 
Authored: Tue Nov 29 15:37:58 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 29 15:37:58 2016 -0800

--
 .../src/main/java/org/apache/beam/sdk/Pipeline.java  | 15 ---
 1 file changed, 15 deletions(-)
--

[1/2] incubator-beam git commit: Remove TransformApplicationsForTesting

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8d127beb8 -> b1f7013d8


Remove TransformApplicationsForTesting

This field is mutated but never queried.

Remove Pipeline#addValueInternal

This method is never called and not suitable for use.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1184bfa7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1184bfa7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1184bfa7

Branch: refs/heads/master
Commit: 1184bfa7a3ee5d58d65c9ba9200e91f71856ce4a
Parents: 8d127be
Author: Thomas Groh 
Authored: Tue Nov 29 14:30:09 2016 -0800
Committer: Thomas Groh 
Committed: Tue Nov 29 15:37:57 2016 -0800

--
 .../src/main/java/org/apache/beam/sdk/Pipeline.java  | 15 ---
 1 file changed, 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1184bfa7/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
index e188b35..9edf496 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
@@ -17,8 +17,6 @@
  */
 package org.apache.beam.sdk;
 
-import com.google.common.collect.HashMultimap;
-import com.google.common.collect.Multimap;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.HashSet;
@@ -329,8 +327,6 @@ public class Pipeline {
   private Collection values = new ArrayList<>();
   private Set usedFullNames = new HashSet<>();
   private CoderRegistry coderRegistry;
-  private Multimap, AppliedPTransform> 
transformApplicationsForTesting =
-  HashMultimap.create();
 
   /**
* @deprecated replaced by {@link #Pipeline(PipelineRunner, PipelineOptions)}
@@ -399,7 +395,6 @@ public class Pipeline {
 
   AppliedPTransform applied = AppliedPTransform.of(
   child.getFullName(), input, output, transform);
-  transformApplicationsForTesting.put(transform, applied);
   // recordAsOutput is a NOOP if already called;
   output.recordAsOutput(applied);
   verifyOutputState(output, child);
@@ -513,14 +508,4 @@ public class Pipeline {
   private String buildName(String namePrefix, String name) {
 return namePrefix.isEmpty() ? name : namePrefix + "/" + name;
   }
-
-  /**
-   * Adds the given {@link PValue} to this {@link Pipeline}.
-   *
-   * For internal use only.
-   */
-  public void addValueInternal(PValue value) {
-this.values.add(value);
-LOG.debug("Adding {} to {}", value, this);
-  }
 }

[GitHub] incubator-beam pull request #1458: Restore a private ValueInEmptyWindows to ...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1458

Restore a private ValueInEmptyWindows to DataflowRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This fixes the postcommit break at 
https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1681/consoleFull
 caused by https://github.com/apache/incubator-beam/pull/1440

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam ISM

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1458.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1458


commit 1d2e6579c55408e1ecb5e3173e3387ef19d6ff30
Author: Kenneth Knowles 
Date:   2016-11-29T23:23:37Z

Restore a private ValueInEmptyWindows to DataflowRunner for special hacks




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1457: Remove TransformApplicationsForTesting

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1457

Remove TransformApplicationsForTesting

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This field is mutated but never queried.

Remove Pipeline#addValueInternal

This method is never called and not suitable for use.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam remove_some_internals

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1457.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1457


commit 9755baf949031a9398049072aa2817f5e82bf8ff
Author: Thomas Groh 
Date:   2016-11-29T22:30:09Z

Remove TransformApplicationsForTesting

This field is mutated but never queried.

Remove Pipeline#addValueInternal

This method is never called and not suitable for use.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1453: Update googledatastore version

[vikkyrk]

Github user vikkyrk closed the pull request at:

https://github.com/apache/incubator-beam/pull/1453


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Update googledatastore version

[dhalperi]

Update googledatastore version


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6c8c17a1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6c8c17a1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6c8c17a1

Branch: refs/heads/python-sdk
Commit: 6c8c17a1c1977ed69860d25dc8ab45640e7a1c53
Parents: ad4dc87
Author: Vikas Kedigehalli 
Authored: Tue Nov 29 09:54:00 2016 -0800
Committer: Dan Halperin 
Committed: Tue Nov 29 14:01:50 2016 -0800

--
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6c8c17a1/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 525f59c..add6dc0 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -87,7 +87,7 @@ REQUIRED_PACKAGES = [
 'avro>=1.7.7,<2.0.0',
 'dill>=0.2.5,<0.3',
 'google-apitools>=0.5.2,<1.0.0',
-'googledatastore==6.4.0',
+'googledatastore>=6.4.1,<7.0.0',
 'httplib2>=0.8,<0.10',
 'mock>=1.0.1,<3.0.0',
 'oauth2client>=2.0.1,<4.0.0',

[1/2] incubator-beam git commit: Closes #1453

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk ad4dc87a4 -> 5ce75a2ea


Closes #1453


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5ce75a2e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5ce75a2e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5ce75a2e

Branch: refs/heads/python-sdk
Commit: 5ce75a2eae31dbab4d07d301716b4d7e3218b8b9
Parents: ad4dc87 6c8c17a
Author: Dan Halperin 
Authored: Tue Nov 29 14:01:50 2016 -0800
Committer: Dan Halperin 
Committed: Tue Nov 29 14:01:50 2016 -0800

--
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] incubator-beam pull request #1456: Testing travis python issue

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1456

Testing travis python issue

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam test-tox-python

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1456.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1456


commit 24f133e9e4c675497b1cb3e9db672db51db180da
Author: Pablo 
Date:   2016-11-29T21:58:55Z

Testing travis python issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1455: Exploring travis issue in Python SDK's li...

[pabloem]

Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1455


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1455: Exploring travis issue in Python SDK's li...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/incubator-beam/pull/1455

Exploring travis issue in Python SDK's linux runs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam test-tox-python

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1455.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1455


commit 009230e8b6e58f2e79d5c6d9a24cb00531d87735
Author: Dan Halperin 
Date:   2016-09-09T20:12:27Z

Closes #933

commit acf530e11d90dc586b5b5f822746ef1bca0ac77c
Author: Charles Chen 
Date:   2016-09-09T21:07:00Z

Use cStringIO instead of StringIO

commit 2649372d6ef10255366e34ee1fcf2f64766b4fd0
Author: Dan Halperin 
Date:   2016-09-10T19:39:13Z

Closes #939

commit 6a483c187dff5d436736f1ff505187c727a95b38
Author: Gus Katsiapis 
Date:   2016-09-09T22:34:39Z

Making Dataflow Python Materialized PCollection representation more
efficient (3 of several).

commit bc32bc8661b31036a879cb2b569b1fe976e2ac4b
Author: Dan Halperin 
Date:   2016-09-13T00:07:22Z

Closes #940

commit f5557c00cd46c8560cad5027678603056a6235ff
Author: Gus Katsiapis 
Date:   2016-09-12T17:11:44Z

Refactoring code in avroio.py to allow for re-use.

* Making sure that _AvroUtils validates the sync_marker.
* This should detect corrupted or not-properly formatted AVRO files.
* Simplifying block reading.
* Running snappy tests only when snappy is installed in the system.

commit 4b584ca26b12f42a21b25b7d72c2e57b20a1e7ad
Author: Dan Halperin 
Date:   2016-09-13T16:59:56Z

Closes #946

commit 4602c954caeda24cc7ce762c2fd9995aa4c04c1a
Author: Charles Chen 
Date:   2016-09-14T19:14:34Z

Insert a shuffle before write finalization

commit a60b58a947fda65cb2f6d8c0bcbbf0756c8407bf
Author: Robert Bradshaw 
Date:   2016-09-14T23:41:14Z

Closes #958

commit 63904e01f52be498ba3c8c9e1989bcffb02dfcd6
Author: Robert Bradshaw 
Date:   2016-09-16T00:00:42Z

Allow pickling of UnwindowedValues instances

These are trivial wrappers produced when "windowing" by the global window, 
but may need to be materialized in some cases.

commit e9b1e41240d5032cdaa2b745c95f94da45475f34
Author: Robert Bradshaw 
Date:   2016-09-16T22:41:22Z

Add unit test for unwindowed iterator picking.

Also lifted this out to a top-level class rather than defining
it on every element now that it's not longer a simple generator
statement.

commit ebae225ed30854296862c46c497d4b3e6e7de9f6
Author: Robert Bradshaw 
Date:   2016-09-16T22:50:09Z

Closes #964

commit 2d1e7ff6d342442f83b97c382f08a03e2bac6572
Author: Chamikara Jayalath 
Date:   2016-08-30T01:08:46Z

Adds a text source to Python SDK.

commit 4b7fe2dc552c838c00e062a644d7d4bc507cf075
Author: Robert Bradshaw 
Date:   2016-09-17T01:01:47Z

Closes #920

commit 24bb8f19329b3d0c1d0330e0c16c41ab1554684d
Author: Marian Dvorsky 
Date:   2016-09-16T17:46:32Z

Removed unnecessary throttling of rename parallelism.

commit 29b55e95600fbf299ac7eb9527ff33bd8030275e
Author: Robert Bradshaw 
Date:   2016-09-20T00:39:47Z

Closes #965

commit 7de9830d96c7928444a5a4849698e70ec423ef62
Author: Christian Hudon 
Date:   2016-09-15T19:10:57Z

Use sys.executable and "-m pip" to ensure we use the same Python and pip as 
the currently running one.

commit 2f09003e33b63594fd87a6cd8bf2803005174fd1
Author: Robert Bradshaw 
Date:   2016-09-20T00:50:58Z

Closes #962

commit 19a8407f741848db0dc86e587ea9739b17888768
Author: Gus Katsiapis 
Date:   2016-09-20T00:21:45Z

Changed ToStringCoder to BytesCoder in test

The former can't be used with sources, as it can only encode.

commit adda1632015043e352c75e2f9966cc04a458b30c
Author: Robert Bradshaw 
Date:   2016-09-20T00:54:27Z

Closes #976

commit 49c0359327ba418cfe62ef2291560d1b1867f4e5
Author: Chamikara Jayalath 
Date:   2016-09-20T05:27:47Z

Updates lint configurations to ignore generated files.

commit b6c7478ff0a17b23b3aa603310b2f5254f350392
Author: Robert Bradshaw 
Date:   2016-09-20T16:10:01Z

Closes #979

commit 2e3384e62ec14f41469c45c3701c1236242dc74c
Author: Chamikara Jayalath 
Dat

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1682

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1681

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706425#comment-15706425
 ] 

ASF GitHub Bot commented on BEAM-551:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1452


> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1452: [BEAM-551] Add a test to demonstrate NVP ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1452


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1452

[dhalperi]

Closes #1452


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8d127beb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8d127beb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8d127beb

Branch: refs/heads/master
Commit: 8d127beb867380b53859c98deba74172db57cc0a
Parents: 4ce85ed 74682c9
Author: Dan Halperin 
Authored: Tue Nov 29 12:22:01 2016 -0800
Committer: Dan Halperin 
Committed: Tue Nov 29 12:22:01 2016 -0800

--
 .../apache/beam/sdk/options/ValueProvider.java|  2 +-
 .../beam/sdk/options/ValueProviderTest.java   | 18 ++
 2 files changed, 19 insertions(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Add a test demonstrating how to use ValueProvider with non-serializable data

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4ce85ed94 -> 8d127beb8


Add a test demonstrating how to use ValueProvider with non-serializable data


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/74682c92
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/74682c92
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/74682c92

Branch: refs/heads/master
Commit: 74682c92d3d2bc5cd5385812ce985a8a75ee4899
Parents: 4ce85ed
Author: Sam McVeety 
Authored: Tue Nov 1 17:58:16 2016 -0700
Committer: Dan Halperin 
Committed: Tue Nov 29 12:22:00 2016 -0800

--
 .../apache/beam/sdk/options/ValueProvider.java|  2 +-
 .../beam/sdk/options/ValueProviderTest.java   | 18 ++
 2 files changed, 19 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/74682c92/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
index 2f52ad4..3a2e7ed 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ValueProvider.java
@@ -51,7 +51,7 @@ import org.apache.beam.sdk.transforms.SerializableFunction;
  */
 @JsonSerialize(using = ValueProvider.Serializer.class)
 @JsonDeserialize(using = ValueProvider.Deserializer.class)
-public interface ValueProvider {
+public interface ValueProvider extends Serializable {
   /**
* Return the value wrapped by this {@link ValueProvider}.
*/

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/74682c92/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
index be0f076..31532b9 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/options/ValueProviderTest.java
@@ -29,6 +29,7 @@ import 
org.apache.beam.sdk.options.ValueProvider.NestedValueProvider;
 import org.apache.beam.sdk.options.ValueProvider.RuntimeValueProvider;
 import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
 import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.util.SerializableUtils;
 import org.junit.Rule;
 import org.junit.Test;
 import org.junit.rules.ExpectedException;
@@ -250,4 +251,21 @@ public class ValueProviderTest {
 expectedException.expectMessage("Not called from a runtime context");
 nvp.get();
   }
+
+  private static class NonSerializable {}
+
+  private static class NonSerializableTranslator
+  implements SerializableFunction {
+@Override
+public NonSerializable apply(String from) {
+  return new NonSerializable();
+}
+  }
+
+  @Test
+  public void testNestedValueProviderSerialize() throws Exception {
+ValueProvider nvp = NestedValueProvider.of(
+StaticValueProvider.of("foo"), new NonSerializableTranslator());
+SerializableUtils.ensureSerializable(nvp);
+  }
 }

[GitHub] incubator-beam pull request #1441: Fix double-close bug

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1441


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Fix double-close bug

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 0d95d8c56 -> 4ce85ed94


Fix double-close bug

The WritableByteChannel returned for GCS locations has a bug where calling 
close twice throws an Exception, so we cannot safely use AutoCloseable here.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/01236906
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/01236906
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/01236906

Branch: refs/heads/master
Commit: 0123690600ebd5f83cf485c92d6a35762428cd84
Parents: 0d95d8c
Author: sammcveety 
Authored: Mon Nov 28 11:26:19 2016 -0800
Committer: Dan Halperin 
Committed: Tue Nov 29 12:19:45 2016 -0800

--
 .../org/apache/beam/runners/dataflow/DataflowRunner.java| 9 +++--
 1 file changed, 3 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/01236906/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 03c503d..641daf4 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -59,7 +59,6 @@ import java.net.URISyntaxException;
 import java.net.URL;
 import java.net.URLClassLoader;
 import java.nio.channels.Channels;
-import java.nio.channels.WritableByteChannel;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collection;
@@ -566,15 +565,13 @@ public class DataflowRunner extends 
PipelineRunner {
   String.format(
   "Location must be local or on Cloud Storage, got {}.", 
fileLocation));
   String workSpecJson = DataflowPipelineTranslator.jobToString(newJob);
-  try (
-  WritableByteChannel writer =
-  IOChannelUtils.create(fileLocation, MimeTypes.TEXT);
-  PrintWriter printWriter = new 
PrintWriter(Channels.newOutputStream(writer))) {
+  try (PrintWriter printWriter = new PrintWriter(
+  Channels.newOutputStream(IOChannelUtils.create(fileLocation, 
MimeTypes.TEXT {
 printWriter.print(workSpecJson);
 LOG.info("Printed job specification to {}", fileLocation);
   } catch (IOException ex) {
 String error =
-String.format("Cannot create output file at {}", fileLocation);
+String.format("Cannot create output file at %s", fileLocation);
 if (isTemplate) {
   throw new RuntimeException(error, ex);
 } else {

[2/2] incubator-beam git commit: Closes #1441

[dhalperi]

Closes #1441


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4ce85ed9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4ce85ed9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4ce85ed9

Branch: refs/heads/master
Commit: 4ce85ed949e6a955433d5ff307cc2af3c38348c8
Parents: 0d95d8c 0123690
Author: Dan Halperin 
Authored: Tue Nov 29 12:19:46 2016 -0800
Committer: Dan Halperin 
Committed: Tue Nov 29 12:19:46 2016 -0800

--
 .../org/apache/beam/runners/dataflow/DataflowRunner.java| 9 +++--
 1 file changed, 3 insertions(+), 6 deletions(-)
--

Jenkins build is back to stable : beam_PostCommit_MavenVerify #1936

[Apache Jenkins Server]

See 

[1/3] incubator-beam git commit: Transmit new DoFn, not OldDoFn, in Dataflow translator

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 1716bfc49 -> 2f86a6ad0


Transmit new DoFn, not OldDoFn, in Dataflow translator


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f52ac3ec
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f52ac3ec
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f52ac3ec

Branch: refs/heads/master
Commit: f52ac3ec75cfec025290f174f0f0529850c2bfd9
Parents: c21167c
Author: Kenneth Knowles 
Authored: Tue Nov 15 22:27:35 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 29 11:07:02 2016 -0800

--
 .../beam/runners/dataflow/DataflowPipelineTranslator.java| 7 +++
 .../org/apache/beam/runners/dataflow/DataflowRunner.java | 8 +++-
 2 files changed, 6 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f52ac3ec/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 0549d5b..2af2cae 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -72,7 +72,6 @@ import org.apache.beam.sdk.transforms.Combine;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
@@ -968,7 +967,7 @@ public class DataflowPipelineTranslator {
 BiMap> outputMap =
 translateOutputs(context.getOutput(transform), context);
 translateFn(
-transform.getFn(),
+transform.getNewFn(),
 context.getInput(transform).getWindowingStrategy(),
 transform.getSideInputs(),
 context.getInput(transform).getCoder(),
@@ -997,7 +996,7 @@ public class DataflowPipelineTranslator {
 translateInputs(context.getInput(transform), 
transform.getSideInputs(), context);
 long mainOutput = context.addOutput(context.getOutput(transform));
 translateFn(
-transform.getFn(),
+transform.getNewFn(),
 context.getInput(transform).getWindowingStrategy(),
 transform.getSideInputs(),
 context.getInput(transform).getCoder(),
@@ -1075,7 +1074,7 @@ public class DataflowPipelineTranslator {
   }
 
   private static void translateFn(
-  OldDoFn fn,
+  DoFn fn,
   WindowingStrategy windowingStrategy,
   Iterable> sideInputs,
   Coder inputCoder,

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f52ac3ec/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index b629d65..ca3f0ed 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -128,7 +128,6 @@ import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.GroupByKey;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SerializableFunction;
@@ -2364,8 +2363,7 @@ public class DataflowRunner extends 
PipelineRunner {
* {@link PCollectionView} backend implementation.
*/
   @Deprecated
-  public static class StreamingPCollectionViewWriterFn
-  extends OldDoFn, T> implements OldDoFn.RequiresWindowAccess {
+  public static class StreamingPCollectionViewWriterFn extends 
DoFn, T> {
 private final PCollectionView view;
 private final Coder dataCoder;
 
@@ -2387,8 +2385,8 @@ public class DataflowRunner extends 
PipelineRunn

[jira] [Commented] (BEAM-230) Remove WindowedValue#valueInEmptyWindows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706247#comment-15706247
 ] 

ASF GitHub Bot commented on BEAM-230:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1440


> Remove WindowedValue#valueInEmptyWindows
> 
>
> Key: BEAM-230
> URL: https://issues.apache.org/jira/browse/BEAM-230
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: Not applicable
>
>
> A WindowedValue in no windows does not exist, and can be dropped by a runner 
> at any time.
> We should also assert that any collection of windows is nonempty when 
> creating a new WindowedValue. If a user wants to drop an element, they should 
> explicitly filter it out rather than expecting it to be dropped by the runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1440: [BEAM-230] Remove WindowedValue.valueInEm...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1440


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: This closes #1440

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 2f86a6ad0 -> 0d95d8c56


This closes #1440


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0d95d8c5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0d95d8c5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0d95d8c5

Branch: refs/heads/master
Commit: 0d95d8c5666d2d00ec7d253ee808476aedaaff6d
Parents: 2f86a6a 0e49b15
Author: Kenneth Knowles 
Authored: Tue Nov 29 11:09:26 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 29 11:09:26 2016 -0800

--
 .../direct/FlattenEvaluatorFactoryTest.java |  8 ++---
 .../beam/runners/dataflow/DataflowRunner.java   | 10 +++---
 .../org/apache/beam/sdk/util/WindowedValue.java | 33 +++-
 .../beam/sdk/testing/PaneExtractorsTest.java|  2 +-
 .../apache/beam/sdk/util/WindowedValueTest.java | 10 --
 5 files changed, 14 insertions(+), 49 deletions(-)
--

[2/2] incubator-beam git commit: Remove WindowedValue.valueInEmptyWindows

[kenn]

Remove WindowedValue.valueInEmptyWindows

A value in empty windows expands to no values, so it can be dropped at
any time, perhaps unintentionally. This has bitten runner authors, including
Spark & Dataflow.

While creating such a thing in memory is not automatically problematic, it
is also not really useful. So this change removes it.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0e49b150
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0e49b150
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0e49b150

Branch: refs/heads/master
Commit: 0e49b150e83d85ae432c640da937a9497068e71b
Parents: 2f86a6a
Author: Kenneth Knowles 
Authored: Tue May 10 11:39:35 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Nov 29 11:09:26 2016 -0800

--
 .../direct/FlattenEvaluatorFactoryTest.java |  8 ++---
 .../beam/runners/dataflow/DataflowRunner.java   | 10 +++---
 .../org/apache/beam/sdk/util/WindowedValue.java | 33 +++-
 .../beam/sdk/testing/PaneExtractorsTest.java|  2 +-
 .../apache/beam/sdk/util/WindowedValueTest.java | 10 --
 5 files changed, 14 insertions(+), 49 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0e49b150/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
index cb27fbc..39c7cab 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/FlattenEvaluatorFactoryTest.java
@@ -78,9 +78,9 @@ public class FlattenEvaluatorFactoryTest {
 rightSideEvaluator.processElement(WindowedValue.valueInGlobalWindow(-1));
 leftSideEvaluator.processElement(
 WindowedValue.timestampedValueInGlobalWindow(2, new Instant(1024)));
-leftSideEvaluator.processElement(WindowedValue.valueInEmptyWindows(4, 
PaneInfo.NO_FIRING));
+leftSideEvaluator.processElement(WindowedValue.valueInGlobalWindow(4, 
PaneInfo.NO_FIRING));
 rightSideEvaluator.processElement(
-WindowedValue.valueInEmptyWindows(2, 
PaneInfo.ON_TIME_AND_ONLY_FIRING));
+WindowedValue.valueInGlobalWindow(2, 
PaneInfo.ON_TIME_AND_ONLY_FIRING));
 rightSideEvaluator.processElement(
 WindowedValue.timestampedValueInGlobalWindow(-4, new Instant(-4096)));
 
@@ -104,12 +104,12 @@ public class FlattenEvaluatorFactoryTest {
 flattenedLeftBundle.commit(Instant.now()).getElements(),
 containsInAnyOrder(
 WindowedValue.timestampedValueInGlobalWindow(2, new Instant(1024)),
-WindowedValue.valueInEmptyWindows(4, PaneInfo.NO_FIRING),
+WindowedValue.valueInGlobalWindow(4, PaneInfo.NO_FIRING),
 WindowedValue.valueInGlobalWindow(1)));
 assertThat(
 flattenedRightBundle.commit(Instant.now()).getElements(),
 containsInAnyOrder(
-WindowedValue.valueInEmptyWindows(2, 
PaneInfo.ON_TIME_AND_ONLY_FIRING),
+WindowedValue.valueInGlobalWindow(2, 
PaneInfo.ON_TIME_AND_ONLY_FIRING),
 WindowedValue.timestampedValueInGlobalWindow(-4, new 
Instant(-4096)),
 WindowedValue.valueInGlobalWindow(-1)));
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0e49b150/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index ca3f0ed..03c503d 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -23,7 +23,7 @@ import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.base.Strings.isNullOrEmpty;
 import static org.apache.beam.sdk.util.StringUtils.approximatePTransformName;
 import static org.apache.beam.sdk.util.StringUtils.approximateSimpleName;
-import static org.apache.beam.sdk.util.WindowedValue.valueInEmptyWindows;
+import static org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
 
 import com.fasterxml.jackson.annotation.JsonCreator;
 import com.fasterxml.jackson.annotation.JsonProperty;
@@ -1233,7 +1233,7 @

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706232#comment-15706232
 ] 

ASF GitHub Bot commented on BEAM-498:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1368


> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1368: [BEAM-498] Transmit new DoFn in Dataflow ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1368


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam git commit: This closes #1368

[kenn]

This closes #1368


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2f86a6ad
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2f86a6ad
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2f86a6ad

Branch: refs/heads/master
Commit: 2f86a6ad0cfc77fc7408bffe3fe74162769b5ac8
Parents: 1716bfc f52ac3e
Author: Kenneth Knowles 
Authored: Tue Nov 29 11:07:03 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 29 11:07:03 2016 -0800

--
 .../runners/dataflow/DataflowPipelineTranslator.java|  7 +++
 .../apache/beam/runners/dataflow/DataflowRunner.java| 12 +---
 2 files changed, 8 insertions(+), 11 deletions(-)
--

[2/3] incubator-beam git commit: Update Dataflow worker to beam-master-20161129

[kenn]

Update Dataflow worker to beam-master-20161129


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c21167ce
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c21167ce
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c21167ce

Branch: refs/heads/master
Commit: c21167ce50875fc3839b858f518aa4995ee92550
Parents: 1716bfc
Author: Kenneth Knowles 
Authored: Mon Nov 28 10:09:39 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 29 11:07:02 2016 -0800

--
 .../java/org/apache/beam/runners/dataflow/DataflowRunner.java| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c21167ce/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index f1d41f2..b629d65 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -212,9 +212,9 @@ public class DataflowRunner extends 
PipelineRunner {
   // Default Docker container images that execute Dataflow worker harness, 
residing in Google
   // Container Registry, separately for Batch and Streaming.
   public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE =
-  "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20161031";
+  "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20161129";
   public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE =
-  "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20161031";
+  "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20161129";
 
   // The limit of CreateJob request size.
   private static final int CREATE_JOB_REQUEST_LIMIT_BYTES = 10 * 1024 * 1024;

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_SparkLocal #261

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify #1935

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1679

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1454: Move datastore example to cookbook

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/1454

Move datastore example to cookbook

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam py_ds_example_move

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1454.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1454






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1453: Update googledatastore version

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/1453

Update googledatastore version

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
- The newer version contains fixes in setup.py which prevents pip from 
overwriting init files from 'google' package, when other google libraries are 
installed. 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam py_ds_upgrade

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1453.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1453


commit cf7cefb101f56e7490ed862272f9b2792f4e46e7
Author: Vikas Kedigehalli 
Date:   2016-11-29T17:54:00Z

Update googledatastore version




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_RunnableOnService_SparkLocal #260

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-551) Support Dynamic PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705995#comment-15705995
 ] 

ASF GitHub Bot commented on BEAM-551:
-

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1452

[BEAM-551] Add a test to demonstrate NVP serialization

R: @dhalperi 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/nvp_repro

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1452.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1452


commit c9a5f8d8bfbe48921b1327acb4d2bfe8fa31f12e
Author: Sam McVeety 
Date:   2016-11-02T00:58:16Z

Demonstrate serializing issue

commit 29932b2fa1798df16d7114ed3dd915e0dcf50d93
Author: Sam McVeety 
Date:   2016-11-29T17:45:10Z

Update test




> Support Dynamic PipelineOptions
> ---
>
> Key: BEAM-551
> URL: https://issues.apache.org/jira/browse/BEAM-551
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sam McVeety
>Assignee: Frances Perry
>Priority: Minor
>
> During the graph construction phase, the given SDK generates an initial
> execution graph for the program.  At execution time, this graph is
> executed, either locally or by a service.  Currently, Beam only supports
> parameterization at graph construction time.  Both Flink and Spark supply
> functionality that allows a pre-compiled job to be run without SDK
> interaction with updated runtime parameters.
> In its current incarnation, Dataflow can read values of PipelineOptions at
> job submission time, but this requires the presence of an SDK to properly
> encode these values into the job.  We would like to build a common layer
> into the Beam model so that these dynamic options can be properly provided
> to jobs.
> Please see
> https://docs.google.com/document/d/1I-iIgWDYasb7ZmXbGBHdok_IK1r1YAJ90JG5Fz0_28o/edit
> for the high-level model, and
> https://docs.google.com/document/d/17I7HeNQmiIfOJi0aI70tgGMMkOSgGi8ZUH-MOnFatZ8/edit
> for
> the specific API proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1452: [BEAM-551] Add a test to demonstrate NVP ...

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1452

[BEAM-551] Add a test to demonstrate NVP serialization

R: @dhalperi 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/nvp_repro

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1452.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1452


commit c9a5f8d8bfbe48921b1327acb4d2bfe8fa31f12e
Author: Sam McVeety 
Date:   2016-11-02T00:58:16Z

Demonstrate serializing issue

commit 29932b2fa1798df16d7114ed3dd915e0dcf50d93
Author: Sam McVeety 
Date:   2016-11-29T17:45:10Z

Update test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705949#comment-15705949
 ] 

ASF GitHub Bot commented on BEAM-25:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1411


> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] incubator-beam git commit: Add isStateful() to DoFnSignature

[kenn]

Add isStateful() to DoFnSignature


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bdd3e086
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bdd3e086
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bdd3e086

Branch: refs/heads/master
Commit: bdd3e0862b91ac682336eb8ff489fff104ea927d
Parents: 3a8b9b5
Author: Kenneth Knowles 
Authored: Mon Nov 28 16:15:55 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 29 09:24:55 2016 -0800

--
 .../org/apache/beam/sdk/transforms/reflect/DoFnSignature.java | 7 +++
 1 file changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bdd3e086/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
index cd93583..0750949 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
@@ -105,6 +105,13 @@ public abstract class DoFnSignature {
   @Nullable
   public abstract Map onTimerMethods();
 
+  /**
+   * Whether the {@link DoFn} described by this signature uses state.
+   */
+  public boolean isStateful() {
+return stateDeclarations().size() > 0;
+  }
+
   static Builder builder() {
 return new AutoValue_DoFnSignature.Builder();
   }

[GitHub] incubator-beam pull request #1411: [BEAM-25] Temporarily reject stateful Par...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1411


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: Reject stateful DoFn in DataflowRunner

[kenn]

Reject stateful DoFn in DataflowRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/31a55f40
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/31a55f40
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/31a55f40

Branch: refs/heads/master
Commit: 31a55f407473f23a61cf6dfe42c3f6f4c7880920
Parents: bdd3e08
Author: Kenneth Knowles 
Authored: Tue Nov 15 21:35:03 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 29 09:24:55 2016 -0800

--
 runners/google-cloud-dataflow-java/pom.xml|  1 +
 .../dataflow/DataflowPipelineTranslator.java  | 18 ++
 2 files changed, 19 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/31a55f40/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index da3a4d6..59276e4 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -77,6 +77,7 @@
   
 runnable-on-service-tests
 
+  
org.apache.beam.sdk.testing.UsesStatefulParDo
   
 org.apache.beam.sdk.transforms.FlattenTest
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/31a55f40/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 9acf071..0549d5b 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -69,6 +69,7 @@ import org.apache.beam.sdk.options.StreamingOptions;
 import org.apache.beam.sdk.runners.TransformTreeNode;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.OldDoFn;
@@ -77,6 +78,7 @@ import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.HasDisplayData;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.util.AppliedCombineFn;
@@ -959,6 +961,8 @@ public class DataflowPipelineTranslator {
   private  void translateMultiHelper(
   ParDo.BoundMulti transform,
   TranslationContext context) {
+rejectStatefulDoFn(transform.getNewFn());
+
 context.addStep(transform, "ParallelDo");
 translateInputs(context.getInput(transform), 
transform.getSideInputs(), context);
 BiMap> outputMap =
@@ -987,6 +991,8 @@ public class DataflowPipelineTranslator {
   private  void translateSingleHelper(
   ParDo.Bound transform,
   TranslationContext context) {
+rejectStatefulDoFn(transform.getNewFn());
+
 context.addStep(transform, "ParallelDo");
 translateInputs(context.getInput(transform), 
transform.getSideInputs(), context);
 long mainOutput = context.addOutput(context.getOutput(transform));
@@ -1033,6 +1039,18 @@ public class DataflowPipelineTranslator {
 registerTransformTranslator(Read.Bounded.class, new ReadTranslator());
   }
 
+  private static void rejectStatefulDoFn(DoFn doFn) {
+if (DoFnSignatures.getSignature(doFn.getClass()).isStateful()) {
+throw new UnsupportedOperationException(
+String.format(
+"Found %s annotations on %s, but %s cannot yet be used with state 
in the %s.",
+DoFn.StateId.class.getSimpleName(),
+doFn.getClass().getName(),
+DoFn.class.getSimpleName(),
+DataflowRunner.class.getSimpleName()));
+}
+  }
+
   private static void translateInputs(
   PCollection input,
   List> sideInputs,

[1/3] incubator-beam git commit: This closes #1411

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3a8b9b521 -> 1716bfc49


This closes #1411


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1716bfc4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1716bfc4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1716bfc4

Branch: refs/heads/master
Commit: 1716bfc4906b97bae434d9740a0af172c91f5e10
Parents: 3a8b9b5 31a55f4
Author: Kenneth Knowles 
Authored: Tue Nov 29 09:24:55 2016 -0800
Committer: Kenneth Knowles 
Committed: Tue Nov 29 09:24:55 2016 -0800

--
 runners/google-cloud-dataflow-java/pom.xml|  1 +
 .../dataflow/DataflowPipelineTranslator.java  | 18 ++
 .../sdk/transforms/reflect/DoFnSignature.java |  7 +++
 3 files changed, 26 insertions(+)
--

[jira] [Commented] (BEAM-506) Fill in the documentation/runners/flink portion of the website

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705589#comment-15705589
 ] 

Maximilian Michels commented on BEAM-506:
-

+1

> Fill in the documentation/runners/flink portion of the website
> --
>
> Key: BEAM-506
> URL: https://issues.apache.org/jira/browse/BEAM-506
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: Aljoscha Krettek
>
> As per 
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
> Should be a landing page for Flink-specific details



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #97: [BEAM-506] Fill in the documentation/r...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/incubator-beam-site/pull/97

[BEAM-506] Fill in the documentation/runners/flink portion of the website

The outline of this is based on the doc for the Cloud Dataflow runner.

R: @davorbonaci and @mxm 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/incubator-beam-site flink-runner-doc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/97.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #97


commit 438c976ad0208bb60e4a29fb6c07659902cc3823
Author: Aljoscha Krettek 
Date:   2016-11-29T15:23:03Z

[BEAM-506] Fill in the documentation/runners/flink portion of the website




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-506) Fill in the documentation/runners/flink portion of the website

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705595#comment-15705595
 ] 

ASF GitHub Bot commented on BEAM-506:
-

GitHub user aljoscha opened a pull request:

https://github.com/apache/incubator-beam-site/pull/97

[BEAM-506] Fill in the documentation/runners/flink portion of the website

The outline of this is based on the doc for the Cloud Dataflow runner.

R: @davorbonaci and @mxm 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/incubator-beam-site flink-runner-doc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/97.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #97


commit 438c976ad0208bb60e4a29fb6c07659902cc3823
Author: Aljoscha Krettek 
Date:   2016-11-29T15:23:03Z

[BEAM-506] Fill in the documentation/runners/flink portion of the website




> Fill in the documentation/runners/flink portion of the website
> --
>
> Key: BEAM-506
> URL: https://issues.apache.org/jira/browse/BEAM-506
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: Aljoscha Krettek
>
> As per 
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
> Should be a landing page for Flink-specific details



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-506) Fill in the documentation/runners/flink portion of the website

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned BEAM-506:
-

Assignee: Aljoscha Krettek  (was: James Malone)

> Fill in the documentation/runners/flink portion of the website
> --
>
> Key: BEAM-506
> URL: https://issues.apache.org/jira/browse/BEAM-506
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: Aljoscha Krettek
>
> As per 
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
> Should be a landing page for Flink-specific details



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1059) Create ExecIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705262#comment-15705262
 ] 

ASF GitHub Bot commented on BEAM-1059:
--

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1451

[BEAM-1059] Create ExecIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-1059-EXECIO

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1451.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1451


commit 2e2ae69b9a2cd2ea2284823d5c0c6bf2d131eda3
Author: Jean-Baptiste Onofré 
Date:   2016-11-28T11:36:51Z

[BEAM-1059] Create ExecIO




> Create ExecIO
> -
>
> Key: BEAM-1059
> URL: https://issues.apache.org/jira/browse/BEAM-1059
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1451: [BEAM-1059] Create ExecIO

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1451

[BEAM-1059] Create ExecIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-1059-EXECIO

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1451.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1451


commit 2e2ae69b9a2cd2ea2284823d5c0c6bf2d131eda3
Author: Jean-Baptiste Onofré 
Date:   2016-11-28T11:36:51Z

[BEAM-1059] Create ExecIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1678

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1059) Create ExecIO

[JIRA]

Jean-Baptiste Onofré created BEAM-1059:
--

 Summary: Create ExecIO
 Key: BEAM-1059
 URL: https://issues.apache.org/jira/browse/BEAM-1059
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1677

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-1052) UnboundedSource splitId uniqueness breaks if more than one source is used.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1052.
-
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> UnboundedSource splitId uniqueness breaks if more than one source is used.
> --
>
> Key: BEAM-1052
> URL: https://issues.apache.org/jira/browse/BEAM-1052
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Aviem Zur
> Fix For: 0.4.0-incubating
>
>
> We use a "running-id" to identify source splits, but we reiterate for each 
> source evaluated.
> Spark already assigns a unique id per InputDStream, it would be unique enough 
> if we replace {{MicrobatchSource}} hash code with one containing both the 
> running-id and the InputDStream id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1052) UnboundedSource splitId uniqueness breaks if more than one source is used.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15704876#comment-15704876
 ] 

ASF GitHub Bot commented on BEAM-1052:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1450


> UnboundedSource splitId uniqueness breaks if more than one source is used.
> --
>
> Key: BEAM-1052
> URL: https://issues.apache.org/jira/browse/BEAM-1052
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Aviem Zur
>
> We use a "running-id" to identify source splits, but we reiterate for each 
> source evaluated.
> Spark already assigns a unique id per InputDStream, it would be unique enough 
> if we replace {{MicrobatchSource}} hash code with one containing both the 
> running-id and the InputDStream id.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1450

[amitsela]

This closes #1450


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3a8b9b52
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3a8b9b52
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3a8b9b52

Branch: refs/heads/master
Commit: 3a8b9b5212972f0128099251884473d06758e2aa
Parents: 3f16f26 a1a4ac0
Author: Sela 
Authored: Tue Nov 29 11:50:51 2016 +0200
Committer: Sela 
Committed: Tue Nov 29 11:50:51 2016 +0200

--
 .../beam/runners/spark/io/MicrobatchSource.java | 20 ++--
 .../beam/runners/spark/io/SourceDStream.java|  3 ++-
 .../spark/stateful/StateSpecFunctions.java  |  2 +-
 3 files changed, 17 insertions(+), 8 deletions(-)
--

[GitHub] incubator-beam pull request #1450: [BEAM-1052] Add InputDStream id to Microb...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1450


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-1052] Add InputDStream id to MicrobatchSource hashcode.

[amitsela]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3f16f2660 -> 3a8b9b521


[BEAM-1052] Add InputDStream id to MicrobatchSource hashcode.

Done to avoid collisions between splits of different sources.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a1a4ac0f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a1a4ac0f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a1a4ac0f

Branch: refs/heads/master
Commit: a1a4ac0fc0376aa4c43a4357f3acc930e2b53c94
Parents: 3f16f26
Author: Aviem Zur 
Authored: Tue Nov 29 09:51:12 2016 +0200
Committer: Sela 
Committed: Tue Nov 29 11:49:31 2016 +0200

--
 .../beam/runners/spark/io/MicrobatchSource.java | 20 ++--
 .../beam/runners/spark/io/SourceDStream.java|  3 ++-
 .../spark/stateful/StateSpecFunctions.java  |  2 +-
 3 files changed, 17 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a1a4ac0f/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
index 4a174aa..5656375 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/MicrobatchSource.java
@@ -54,6 +54,7 @@ public class MicrobatchSource(splits.get(i), maxReadTime, 1, 
numRecords[i], i));
+  result.add(new MicrobatchSource<>(splits.get(i), maxReadTime, 1, 
numRecords[i], i, sourceId));
 }
 return result;
   }
@@ -137,8 +140,8 @@ public class MicrobatchSource that = (MicrobatchSource) o;
-
+if (sourceId != that.sourceId) {
+  return false;
+}
 return splitId == that.splitId;
+
   }
 
   @Override
   public int hashCode() {
-return splitId;
+int result = sourceId;
+result = 31 * result + splitId;
+return result;
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a1a4ac0f/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceDStream.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceDStream.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceDStream.java
index 4e47757..84b247b 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceDStream.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SourceDStream.java
@@ -83,7 +83,8 @@ public class SourceDStream, CheckpointMarkT>>> compute(Time 
validTime) {
 MicrobatchSource microbatchSource = new 
MicrobatchSource<>(
-unboundedSource, boundReadDuration, initialParallelism, 
rateControlledMaxRecords(), -1);
+unboundedSource, boundReadDuration, initialParallelism, 
rateControlledMaxRecords(), -1,
+id());
 RDD, CheckpointMarkT>> rdd = new 
SourceRDD.Unbounded<>(
 ssc().sc(), runtimeContext, microbatchSource);
 return scala.Option.apply(rdd);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a1a4ac0f/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/StateSpecFunctions.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/StateSpecFunctions.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/StateSpecFunctions.java
index 48849c2..053f4ac 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/StateSpecFunctions.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/stateful/StateSpecFunctions.java
@@ -144,7 +144,7 @@ public class StateSpecFunctions {
 
   // close and checkpoint reader.
   reader.close();
-  LOG.info("Source id {} spent {} msec on reading.", 
microbatchSource.getSplitId(),
+  LOG.info("Source id {} spent {} msec on reading.", 
microbatchSource.getId(),
   stopwatch.stop().elapsed(TimeUnit.MILLISECONDS));
 
   // if the Source does not supply a CheckpointMark skip updating the 
state.

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1676

[Apache Jenkins Server]

See