[jira] [Commented] (BEAM-1185) Remove the word Pipeline from the name of all PipelineRunner implementations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867404#comment-15867404
 ] 

ASF GitHub Bot commented on BEAM-1185:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2010


> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-1185
> URL: https://issues.apache.org/jira/browse/BEAM-1185
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>  Labels: backward-incompatible, sdk-consistency
> Fix For: Not applicable
>
>
> See: https://issues.apache.org/jira/browse/BEAM-234
> Rename all runners to remove the Pipeline word from their name in the Python 
> SDK (e.g. DirectPipelineRunner -> DirectRunner).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2010: [BEAM-1185] Make examples blocking as command line ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2010


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2010

[altay]

This closes #2010


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e720a7c4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e720a7c4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e720a7c4

Branch: refs/heads/master
Commit: e720a7c43b12347d738c24baa9010d5b877625fd
Parents: db19c7d 2fadeb8
Author: Ahmet Altay 
Authored: Tue Feb 14 23:48:44 2017 -0800
Committer: Ahmet Altay 
Committed: Tue Feb 14 23:48:44 2017 -0800

--
 .../apache_beam/examples/complete/juliaset/juliaset/juliaset.py| 2 +-
 .../examples/complete/juliaset/juliaset/juliaset_test.py   | 2 +-
 sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py| 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
--

[1/2] beam git commit: Make examples blocking as command line invoked

[altay]

Repository: beam
Updated Branches:
  refs/heads/master db19c7df5 -> e720a7c43


Make examples blocking as command line invoked


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2fadeb82
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2fadeb82
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2fadeb82

Branch: refs/heads/master
Commit: 2fadeb8205ec61c11dda5b623f28de688f6c406b
Parents: db19c7d
Author: Sourabh Bajaj 
Authored: Tue Feb 14 17:15:04 2017 -0800
Committer: Ahmet Altay 
Committed: Tue Feb 14 23:48:23 2017 -0800

--
 .../apache_beam/examples/complete/juliaset/juliaset/juliaset.py| 2 +-
 .../examples/complete/juliaset/juliaset/juliaset_test.py   | 2 +-
 sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py| 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2fadeb82/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset.py
--
diff --git 
a/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset.py 
b/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset.py
index 30883dc..8e5d5b3 100644
--- a/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset.py
+++ b/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset.py
@@ -113,7 +113,7 @@ def run(argv=None):  # pylint: disable=missing-docstring
lambda (k, coords): ' '.join('(%s, %s, %s)' % coord for coord in 
coords))
| WriteToText(known_args.coordinate_output))
   # pylint: enable=expression-not-assigned
-  return p.run()
+  return p.run().wait_until_finish()
 
   # Optionally render the image and save it to a file.
   # TODO(silviuc): Add this functionality.

http://git-wip-us.apache.org/repos/asf/beam/blob/2fadeb82/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py
--
diff --git 
a/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py 
b/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py
index c254eb4..c13e857 100644
--- 
a/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py
+++ 
b/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py
@@ -52,7 +52,7 @@ class JuliaSetTest(unittest.TestCase):
 if image_file_name is not None:
   args.append('--image_output=%s' % image_file_name)
 
-juliaset.run(args).wait_until_finish()
+juliaset.run(args)
 
   def test_output_file_format(self):
 grid_size = 5

http://git-wip-us.apache.org/repos/asf/beam/blob/2fadeb82/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py
--
diff --git a/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py 
b/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py
index 6d79216..1eade9d 100644
--- a/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py
+++ b/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py
@@ -90,7 +90,7 @@ def run(argv=None):
   write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE))
 
   # Run the pipeline (all operations are deferred until run() is called).
-  p.run()
+  p.run().wait_until_finish()
 
 
 if __name__ == '__main__':

[1/3] beam-site git commit: Fix typo in stateful processing post

[kenn]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 00e375003 -> f3c189568


Fix typo in stateful processing post


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/47b1c5cd
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/47b1c5cd
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/47b1c5cd

Branch: refs/heads/asf-site
Commit: 47b1c5cdaef819bc82fb6ec069a3a5d1c3e9165c
Parents: 00e3750
Author: Kenneth Knowles 
Authored: Tue Feb 14 21:13:48 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 21:13:48 2017 -0800

--
 src/_posts/2017-02-13-stateful-processing.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/47b1c5cd/src/_posts/2017-02-13-stateful-processing.md
--
diff --git a/src/_posts/2017-02-13-stateful-processing.md 
b/src/_posts/2017-02-13-stateful-processing.md
index 28aee12..b00361a 100644
--- a/src/_posts/2017-02-13-stateful-processing.md
+++ b/src/_posts/2017-02-13-stateful-processing.md
@@ -25,7 +25,7 @@ what you might use it for, and what it looks like in code.
 
 First, a quick recap: In Beam, a big data processing _pipeline_ is a directed,
 acyclic graph of parallel operations called _`PTransforms`_ processing data
-from _`PCollections`_ I'll expand on that by walking through this illustration:
+from _`PCollections`_. I'll expand on that by walking through this 
illustration:
 
 

[GitHub] beam-site pull request #155: Fix typo in stateful processing post

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/155


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam-site git commit: This closes #155: Fix typo in stateful processing post

[kenn]

This closes #155: Fix typo in stateful processing post

  Regenerate site
  Fix typo in stateful processing post


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/f3c18956
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/f3c18956
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/f3c18956

Branch: refs/heads/asf-site
Commit: f3c189568ed5a1a9e0fb84132cf508e271e4cb63
Parents: 00e3750 7602168
Author: Kenneth Knowles 
Authored: Tue Feb 14 21:29:34 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 21:29:34 2017 -0800

--
 content/blog/2017/02/13/stateful-processing.html | 2 +-
 content/feed.xml | 2 +-
 src/_posts/2017-02-13-stateful-processing.md | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
--

[2/3] beam-site git commit: Regenerate site

[kenn]

Regenerate site


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/76021685
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/76021685
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/76021685

Branch: refs/heads/asf-site
Commit: 7602168530feee8aa6eedfd20f5549617b4545e6
Parents: 47b1c5c
Author: Kenneth Knowles 
Authored: Tue Feb 14 21:29:15 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 21:29:15 2017 -0800

--
 content/blog/2017/02/13/stateful-processing.html | 2 +-
 content/feed.xml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/76021685/content/blog/2017/02/13/stateful-processing.html
--
diff --git a/content/blog/2017/02/13/stateful-processing.html 
b/content/blog/2017/02/13/stateful-processing.html
index dcf077a..833b2fd 100644
--- a/content/blog/2017/02/13/stateful-processing.html
+++ b/content/blog/2017/02/13/stateful-processing.html
@@ -175,7 +175,7 @@ the current status in each runner.
 
 First, a quick recap: In Beam, a big data processing pipeline is a 
directed,
 acyclic graph of parallel operations called PTransforms processing data
-from PCollections I’ll 
expand on that by walking through this illustration:
+from PCollections. I’ll 
expand on that by walking through this illustration:
 
 
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/76021685/content/feed.xml
--
diff --git a/content/feed.xml b/content/feed.xml
index 10bccbd..726cdd0 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -29,7 +29,7 @@ the current status in each runner.

 
 
First, a quick recap: In Beam, a big data processing 
pipeline is a directed,
 acyclic graph of parallel operations called PTransforms 
processing data
-from PCollections 
I’ll expand on that by walking through this illustration:
+from PCollections. 
I’ll expand on that by walking through this illustration:

 
 

 

[GitHub] beam-site pull request #155: Fix typo in stateful processing post

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam-site/pull/155

Fix typo in stateful processing post



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam-site typo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/155.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #155


commit 47b1c5cdaef819bc82fb6ec069a3a5d1c3e9165c
Author: Kenneth Knowles 
Date:   2017-02-15T05:13:48Z

Fix typo in stateful processing post




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1490) master branch for apache/beam-site still contains the Beam code

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1490:
-

 Summary: master branch for apache/beam-site still contains the 
Beam code
 Key: BEAM-1490
 URL: https://issues.apache.org/jira/browse/BEAM-1490
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Kenneth Knowles
Assignee: Davor Bonaci
Priority: Trivial


I accidentally typed {{git checkout -b mybranch github/master}} today instead 
of {{git checkout -b mybranch github/asf-site}} and I was dropped into a really 
old Beam codebase.

Perhaps the best thing to have here would be just one empty file called 
{{you-probably-meant-to-checkout-asf-site}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867257#comment-15867257
 ] 

ASF GitHub Bot commented on BEAM-115:
-

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/2000


> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2000: [BEAM-115,BEAM-1348] Unify Fn API and Runner API Fu...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/2000


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1195) Give triggers a cross-language serialization schema

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867254#comment-15867254
 ] 

ASF GitHub Bot commented on BEAM-1195:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1988


> Give triggers a cross-language serialization schema
> ---
>
> Key: BEAM-1195
> URL: https://issues.apache.org/jira/browse/BEAM-1195
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> We have recently gotten to the point where triggers are just syntax, but it 
> is still shipped via Java serialization. To make it language-independent, we 
> need a concrete syntax.
> Something like the following is fairly concise, tag adjacent to payload. I 
> haven't bothered making up fully verbose/namespaced URNs here.
> {code}
> {
> "$urn": "OrFinally",
> "main": {
>   "$urn": "EndOfWindow",
>   "early": 
> },
> "finally": {
>   "$urn": "AfterCount",
>   "count": 45
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[8/9] beam git commit: Upgrade Dataflow container version to beam-master-20170214

[kenn]

Upgrade Dataflow container version to beam-master-20170214


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e097b757
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e097b757
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e097b757

Branch: refs/heads/master
Commit: e097b7575c0be830cf966bd179b3268eb463659e
Parents: b6e9f73
Author: Kenneth Knowles 
Authored: Tue Feb 14 19:30:56 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 19:30:56 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e097b757/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index d366ddc..ec1a927 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20170208
+
beam-master-20170214
 6
   
 

[GitHub] beam pull request #1988: [BEAM-1195, BEAM-115] Add full-fidelity conversion ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1988


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/9] beam git commit: Upgrade Java triggers to support runner API deserialization

[kenn]

Upgrade Java triggers to support runner API deserialization


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5d5602db
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5d5602db
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5d5602db

Branch: refs/heads/master
Commit: 5d5602dbff41ef48add2ea763527f8c0901f0bc0
Parents: 40c4a5c
Author: Kenneth Knowles 
Authored: Sun Feb 12 15:53:17 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 14:55:49 2017 -0800

--
 .../AfterSynchronizedProcessingTimeStateMachine.java  |  6 +-
 .../beam/runners/core/triggers/TriggerStateMachines.java  |  2 +-
 .../AfterSynchronizedProcessingTimeStateMachineTest.java  |  3 ++-
 .../apache/beam/sdk/transforms/windowing/AfterAll.java|  7 +++
 .../apache/beam/sdk/transforms/windowing/AfterEach.java   |  7 +++
 .../apache/beam/sdk/transforms/windowing/AfterFirst.java  |  7 +++
 .../windowing/AfterSynchronizedProcessingTime.java| 10 +++---
 .../windowing/AfterSynchronizedProcessingTimeTest.java|  2 +-
 8 files changed, 37 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/5d5602db/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
index 1319a13..07fab22 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
@@ -28,13 +28,17 @@ import org.joda.time.Instant;
 // This should not really have the superclass 
https://issues.apache.org/jira/browse/BEAM-1486
 class AfterSynchronizedProcessingTimeStateMachine extends 
AfterDelayFromFirstElementStateMachine {
 
+  public static AfterSynchronizedProcessingTimeStateMachine ofFirstElement() {
+return new AfterSynchronizedProcessingTimeStateMachine();
+  }
+
   @Override
   @Nullable
   public Instant getCurrentTime(TriggerStateMachine.TriggerContext context) {
 return context.currentSynchronizedProcessingTime();
   }
 
-  public AfterSynchronizedProcessingTimeStateMachine() {
+  private AfterSynchronizedProcessingTimeStateMachine() {
 super(TimeDomain.SYNCHRONIZED_PROCESSING_TIME,
 Collections.>emptyList());
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/5d5602db/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachines.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachines.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachines.java
index f0e9d21..b13ac40 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachines.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachines.java
@@ -116,7 +116,7 @@ public class TriggerStateMachines {
 }
 
 private OnceTriggerStateMachine 
evaluateSpecific(AfterSynchronizedProcessingTime v) {
-  return new AfterSynchronizedProcessingTimeStateMachine();
+  return AfterSynchronizedProcessingTimeStateMachine.ofFirstElement();
 }
 
 private OnceTriggerStateMachine evaluateSpecific(AfterFirst v) {

http://git-wip-us.apache.org/repos/asf/beam/blob/5d5602db/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachineTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachineTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachineTest.java
index 140bd62..7bfd48d 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachineTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachineTest.java
@@ -36,7 +36,8 @@ import org.junit.runners.JUnit4;
 @RunWith(JUnit4.class)
 public class AfterSynchronizedProcessingTimeStateMachineTest {
 
-  private TriggerStateMachine underTest = new 
AfterSynchronizedProcessingTimeStateMachine();
+  private TriggerStateMachine underTe

[3/9] beam git commit: Fix typo in runner API generated Java class

[kenn]

Fix typo in runner API generated Java class


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/67854e66
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/67854e66
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/67854e66

Branch: refs/heads/master
Commit: 67854e66e97889b3f3e03de297e5af5b73c3fab1
Parents: 5d5602d
Author: Kenneth Knowles 
Authored: Sat Feb 11 16:48:05 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 14:55:49 2017 -0800

--
 sdks/common/runner-api/src/main/proto/beam_runner_api.proto | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/67854e66/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
--
diff --git a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto 
b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
index 195ce01..370b57c 100644
--- a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
+++ b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
@@ -25,7 +25,7 @@ syntax = "proto3";
 
 package org.apache.beam.runner_api.v1;
 
-option java_package = "org.apache.beam.sdks.common.runner_api.v1";
+option java_package = "org.apache.beam.sdk.common.runner_api.v1";
 option java_outer_classname = "RunnerApi";
 
 import "google/protobuf/any.proto";

[2/9] beam git commit: Flesh out triggers in Runner API proto

[kenn]

Flesh out triggers in Runner API proto


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/661cd8d7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/661cd8d7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/661cd8d7

Branch: refs/heads/master
Commit: 661cd8d7407d5f414d5d94badacdeadb519107b7
Parents: 67854e6
Author: Kenneth Knowles 
Authored: Sat Feb 11 17:32:21 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 14:55:49 2017 -0800

--
 .../src/main/proto/beam_runner_api.proto| 109 ++-
 1 file changed, 83 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/661cd8d7/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
--
diff --git a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto 
b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
index 370b57c..91f1558 100644
--- a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
+++ b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
@@ -29,7 +29,6 @@ option java_package = 
"org.apache.beam.sdk.common.runner_api.v1";
 option java_outer_classname = "RunnerApi";
 
 import "google/protobuf/any.proto";
-import "google/protobuf/timestamp.proto";
 
 // A Pipeline is a hierarchical graph of PTransforms, linked
 // by PCollections.
@@ -402,6 +401,24 @@ enum OutputTime {
   EARLIEST_IN_PANE = 2;
 }
 
+// The different time domains in the Beam model.
+enum TimeDomain {
+
+  // Event time is time from the perspective of the data
+  EVENT_TIME = 0;
+
+  // Processing time is time from the perspective of the
+  // execution of your pipeline
+  PROCESSING_TIME = 1;
+
+  // Synchronized processing time is the minimum of the
+  // processing time of all pending elements.
+  //
+  // The "processing time" of an element refers to
+  // the local processing time at which it was emitted
+  SYNCHRONIZED_PROCESSING_TIME = 2;
+}
+
 // A small DSL for expressing when to emit new aggregations
 // from a GroupByKey or CombinePerKey
 //
@@ -439,27 +456,31 @@ message Trigger {
   }
 
   // After input arrives, ready when the specified delay has passed.
-  message AfterProcessingTimeDelay {
-// (Required) The delay, in milliseconds.
-int64 delay_millis = 1;
+  message AfterProcessingTime {
+
+// (Required) The transforms to apply to an arriving element's timestamp,
+// in order
+repeated TimestampTransform timestamp_transforms = 1;
   }
 
-  // After input arrives, ready when the synchronized processing time
-  // progresses as far as the given delay.
-  message AfterSynchronizedProcessingTimeDelay {
-// (Required) The delay, in milliseconds.
-int64 delay_millis = 1;
+  // Ready whenever upstream processing time has all caught up with
+  // the arrival time of an input element
+  message AfterSynchronizedProcessingTime {
+  }
+
+  // The default trigger. Equivalent to Repeat { AfterEndOfWindow } but
+  // specially denoted to indicate the user did not alter the triggering.
+  message Default {
+  }
+
+  // Ready whenever the requisite number of input elements have arrived
+  message ElementCount {
+int32 element_count = 1;
   }
 
   // Never ready. There will only be an ON_TIME output and a final
   // output at window expiration.
-  message Never { }
-
-  // Ready whenever the subtrigger is ready; resets state when the subtrigger
-  // completes.
-  message Repeat {
-// (Require) Trigger that is run repeatedly.
-Trigger subtrigger = 1;
+  message Never {
   }
 
   // Ready whenever either of its subtriggers are ready, but finishes output
@@ -473,9 +494,12 @@ message Trigger {
 Trigger finally = 2;
   }
 
-  // The default trigger. Equivalent to Repeat { AfterEndOfWindow } but
-  // specially denoted to indicate the user did not alter the triggering.
-  message Default { }
+  // Ready whenever the subtrigger is ready; resets state when the subtrigger
+  // completes.
+  message Repeat {
+// (Require) Trigger that is run repeatedly.
+Trigger subtrigger = 1;
+  }
 
   // The full disjoint union of possible triggers.
   oneof trigger {
@@ -483,12 +507,39 @@ message Trigger {
 AfterAny after_any = 2;
 AfterEach after_each = 3;
 AfterEndOfWindow after_end_of_widow = 4;
-AfterProcessingTimeDelay after_processing_time_delay = 5;
-AfterSynchronizedProcessingTimeDelay 
after_synchronized_processing_time_delay = 6;
-Never never = 7;
-Repeat repeat = 8;
-OrFinally or_finally = 9;
-Default default = 10;
+AfterProcessingTime after_processing_time = 5;
+AfterSynchronizedProcessingTime after_synchronized_processing_time = 6;
+Default default = 7;
+ElementCount element_count = 8;
+Never never = 9;
+OrFinally or_f

[7/9] beam git commit: Remove underscore from Runner API proto Java package

[kenn]

Remove underscore from Runner API proto Java package


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b6e9f73a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b6e9f73a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b6e9f73a

Branch: refs/heads/master
Commit: b6e9f73ae00813167c34b55459b9832e20d9aa41
Parents: f4ceaee
Author: Kenneth Knowles 
Authored: Tue Feb 14 12:33:43 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 14:55:50 2017 -0800

--
 sdks/common/runner-api/src/main/proto/beam_runner_api.proto | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b6e9f73a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
--
diff --git a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto 
b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
index 91f1558..a9133ab 100644
--- a/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
+++ b/sdks/common/runner-api/src/main/proto/beam_runner_api.proto
@@ -25,7 +25,7 @@ syntax = "proto3";
 
 package org.apache.beam.runner_api.v1;
 
-option java_package = "org.apache.beam.sdk.common.runner_api.v1";
+option java_package = "org.apache.beam.sdk.common.runner.v1";
 option java_outer_classname = "RunnerApi";
 
 import "google/protobuf/any.proto";

[5/9] beam git commit: Add conversion to/from Java SDK trigger to runner API proto

[kenn]

Add conversion to/from Java SDK trigger to runner API proto


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f4ceaeef
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f4ceaeef
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f4ceaeef

Branch: refs/heads/master
Commit: f4ceaeefe9e8e9d069b760e166c7057a00465360
Parents: 2803864
Author: Kenneth Knowles 
Authored: Sat Feb 11 17:50:27 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 14:55:49 2017 -0800

--
 .../beam/sdk/transforms/windowing/Triggers.java | 313 +++
 .../sdk/transforms/windowing/TriggersTest.java  | 100 ++
 2 files changed, 413 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f4ceaeef/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Triggers.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Triggers.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Triggers.java
new file mode 100644
index 000..8ac904c
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Triggers.java
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.transforms.windowing;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.io.Serializable;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.common.runner.v1.RunnerApi;
+import 
org.apache.beam.sdk.transforms.windowing.AfterWatermark.AfterWatermarkEarlyAndLate;
+import org.apache.beam.sdk.transforms.windowing.Never.NeverTrigger;
+import org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger;
+import org.apache.beam.sdk.util.TimeDomain;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/** Utilities for working with {@link Triggers Triggers}. */
+@Experimental(Experimental.Kind.TRIGGER)
+public class Triggers implements Serializable {
+
+  @VisibleForTesting static final ProtoConverter CONVERTER = new 
ProtoConverter();
+
+  public static RunnerApi.Trigger toProto(Trigger trigger) {
+return CONVERTER.convertTrigger(trigger);
+  }
+
+  @VisibleForTesting
+  static class ProtoConverter {
+
+public RunnerApi.Trigger convertTrigger(Trigger trigger) {
+  Method evaluationMethod = getEvaluationMethod(trigger.getClass());
+  return tryConvert(evaluationMethod, trigger);
+}
+
+private RunnerApi.Trigger tryConvert(Method evaluationMethod, Trigger 
trigger) {
+  try {
+return (RunnerApi.Trigger) evaluationMethod.invoke(this, trigger);
+  } catch (InvocationTargetException exc) {
+if (exc.getCause() instanceof RuntimeException) {
+  throw (RuntimeException) exc.getCause();
+} else {
+  throw new RuntimeException(exc.getCause());
+}
+  } catch (IllegalAccessException exc) {
+throw new IllegalStateException(
+String.format("Internal error: could not invoke %s", 
evaluationMethod));
+  }
+}
+
+private Method getEvaluationMethod(Class clazz) {
+  try {
+return getClass().getDeclaredMethod("convertSpecific", clazz);
+  } catch (NoSuchMethodException exc) {
+throw new IllegalArgumentException(
+String.format(
+"Cannot translate trigger class %s to a runner-API proto.",
+clazz.getCanonicalName()),
+exc);
+  }
+}
+
+private RunnerApi.Trigger convertSpecific(DefaultTrigger v) {
+  return RunnerApi.Trigger.newBuilder()
+  .setDefault(RunnerApi.Trigger.Default.getDefaultInstance())
+  .build();
+}
+
+private RunnerApi.Trigger convertSpecific(AfterWatermark.FromEndOfWindow 
v) {
+  return RunnerApi.Trigger.newBuilder()
+  .setAf

[1/9] beam git commit: Reify delay and alignment in AfterProcessingTime transform

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 9061c65e3 -> db19c7df5


Reify delay and alignment in AfterProcessingTime transform


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/40c4a5cb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/40c4a5cb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/40c4a5cb

Branch: refs/heads/master
Commit: 40c4a5cb6eaa0350a26fe1f215eb812541a7b105
Parents: bea101a
Author: Kenneth Knowles 
Authored: Sun Feb 12 15:03:48 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 14:55:48 2017 -0800

--
 .../AfterDelayFromFirstElementStateMachine.java |   4 +-
 .../AfterProcessingTimeStateMachine.java|   2 +
 ...rSynchronizedProcessingTimeStateMachine.java |   1 +
 .../core/triggers/TriggerStateMachines.java |  36 ++-
 .../core/triggers/TriggerStateMachinesTest.java |   7 +-
 .../windowing/AfterDelayFromFirstElement.java   | 240 ---
 .../windowing/AfterProcessingTime.java  | 105 ++--
 .../AfterSynchronizedProcessingTime.java|  25 +-
 .../windowing/TimestampTransform.java   |  64 +
 .../windowing/AfterProcessingTimeTest.java  |   2 +-
 10 files changed, 192 insertions(+), 294 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/40c4a5cb/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
index b720644..c22 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
@@ -47,6 +47,8 @@ import org.joda.time.format.PeriodFormatter;
  *
  * This class is for internal use only and may change at any time.
  */
+// This class should be inlined to subclasses and deleted, simplifying them too
+// https://issues.apache.org/jira/browse/BEAM-1486
 @Experimental(Experimental.Kind.TRIGGER)
 public abstract class AfterDelayFromFirstElementStateMachine extends 
OnceTriggerStateMachine {
 
@@ -250,7 +252,7 @@ public abstract class 
AfterDelayFromFirstElementStateMachine extends OnceTrigger
   /**
* A {@link SerializableFunction} to delay the timestamp at which this 
triggers fires.
*/
-  private static final class DelayFn implements SerializableFunction {
+  static final class DelayFn implements SerializableFunction 
{
 private final Duration delay;
 
 public DelayFn(Duration delay) {

http://git-wip-us.apache.org/repos/asf/beam/blob/40c4a5cb/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterProcessingTimeStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterProcessingTimeStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterProcessingTimeStateMachine.java
index 2490463..eaf5613 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterProcessingTimeStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterProcessingTimeStateMachine.java
@@ -34,6 +34,8 @@ import org.joda.time.Instant;
  * AfterDelayFromFirstElementStateMachine#plusDelayOf} or {@link
  * AfterDelayFromFirstElementStateMachine#alignedTo}.
  */
+// The superclass should be inlined here, its only real use
+// https://issues.apache.org/jira/browse/BEAM-1486
 @Experimental(Experimental.Kind.TRIGGER)
 public class AfterProcessingTimeStateMachine extends 
AfterDelayFromFirstElementStateMachine {
 

http://git-wip-us.apache.org/repos/asf/beam/blob/40c4a5cb/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
index 000f6e7..1319a13 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterSynchronizedProcessingTimeStateMachine.java
@@ -25,6 +25,7 @@ import org.apache

[9/9] beam git commit: This resolves #1988: Add full-fidelity conversion to/from Java SDK triggers and Runner API protos

[kenn]

This resolves #1988: Add full-fidelity conversion to/from Java SDK triggers and 
Runner API protos

  Upgrade Dataflow container version to beam-master-20170214
  Remove underscore from Runner API proto Java package
  Add conversion to/from Java SDK trigger to runner API proto
  Add runner API config to poms
  Flesh out triggers in Runner API proto
  Fix typo in runner API generated Java class
  Upgrade Java triggers to support runner API deserialization
  Reify delay and alignment in AfterProcessingTime transform


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/db19c7df
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/db19c7df
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/db19c7df

Branch: refs/heads/master
Commit: db19c7df50294dd5697822d3f4038211797383d7
Parents: 9061c65 e097b75
Author: Kenneth Knowles 
Authored: Tue Feb 14 20:51:00 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 20:51:00 2017 -0800

--
 pom.xml |   8 +
 .../AfterDelayFromFirstElementStateMachine.java |   4 +-
 .../AfterProcessingTimeStateMachine.java|   2 +
 ...rSynchronizedProcessingTimeStateMachine.java |   7 +-
 .../core/triggers/TriggerStateMachines.java |  38 ++-
 ...chronizedProcessingTimeStateMachineTest.java |   3 +-
 .../core/triggers/TriggerStateMachinesTest.java |   7 +-
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../src/main/proto/beam_runner_api.proto| 111 +--
 sdks/java/core/pom.xml  |   5 +
 .../beam/sdk/transforms/windowing/AfterAll.java |   7 +
 .../windowing/AfterDelayFromFirstElement.java   | 240 --
 .../sdk/transforms/windowing/AfterEach.java |   7 +
 .../sdk/transforms/windowing/AfterFirst.java|   7 +
 .../windowing/AfterProcessingTime.java  | 105 +--
 .../AfterSynchronizedProcessingTime.java|  31 +-
 .../windowing/TimestampTransform.java   |  64 
 .../beam/sdk/transforms/windowing/Triggers.java | 313 +++
 .../windowing/AfterProcessingTimeTest.java  |   2 +-
 .../AfterSynchronizedProcessingTimeTest.java|   2 +-
 .../sdk/transforms/windowing/TriggersTest.java  | 100 ++
 21 files changed, 738 insertions(+), 327 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/db19c7df/runners/google-cloud-dataflow-java/pom.xml
--

[6/9] beam git commit: Add runner API config to poms

[kenn]

Add runner API config to poms


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2803864b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2803864b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2803864b

Branch: refs/heads/master
Commit: 2803864ba689df92c993b0c4afc392df4558b6bf
Parents: 661cd8d
Author: Kenneth Knowles 
Authored: Sat Feb 11 16:47:15 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 14:55:49 2017 -0800

--
 pom.xml| 8 
 sdks/java/core/pom.xml | 5 +
 2 files changed, 13 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2803864b/pom.xml
--
diff --git a/pom.xml b/pom.xml
index be75659..d53502e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -320,6 +320,7 @@
 beam-sdks-common-fn-api
 ${project.version}
   
+
   
 org.apache.beam
 beam-sdks-common-fn-api
@@ -329,6 +330,13 @@
 
   
 org.apache.beam
+beam-sdks-common-runner-api
+${project.version}
+  
+
+
+  
+org.apache.beam
 beam-runners-flink_2.10-examples
 ${project.version}
   

http://git-wip-us.apache.org/repos/asf/beam/blob/2803864b/sdks/java/core/pom.xml
--
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index 266e144..4f89550 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -183,6 +183,11 @@
 
   
 
+  org.apache.beam
+  beam-sdks-common-runner-api
+
+
+
   io.grpc
   grpc-auth
 

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867182#comment-15867182
 ] 

ASF GitHub Bot commented on BEAM-115:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2011

[BEAM-115,BEAM-1348] Unify Fn API and Runner API coder specs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This includes the entirety of #2000, which unified `FunctionSpec`. Feel 
free to review that first or, if you prefer, review just this since the diff is 
small.

The unification here was uninteresting on top of #2000. Summary of changes:

 - Moved a coder's local id out of the `Coder` itself and into the key of a 
map in `ProcessBundleDescriptor`. Philosophically, the id is an essential 
aspect of `ProcessBundleDescriptor` (or `Pipeline`) but not not an essential 
aspect of a coder.  Pragmatically, this allows the Runner API and the Fn API to 
key the map on different types (`string` and `int64` respectively). 
Prospectively, it makes it easy to construct instances of the message that are 
"just values" without any id, which is aesthetically pleasing and more flexible 
to more uses.
 - Inlined the `SdkFunctionSpec` in `Coder` in the Runner API. Having it by 
reference introduces a needless sharing of key type and adds needless overhead, 
since coders are already stored by reference, as are environments.

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam fn-api-coders

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2011


commit 2b55a7f303ab0fea58ad279dd214253b4fe69565
Author: Kenneth Knowles 
Date:   2017-02-14T20:33:43Z

Remove underscore from Runner API proto Java package

commit 4e7865b828eae962532f1759833eed8b0e769cc9
Author: Kenneth Knowles 
Date:   2017-02-13T16:38:40Z

Unify Fn API and Runner API FunctionSpec

commit 5b5e6290e893385c47799cf5523c29be64c102fd
Author: Kenneth Knowles 
Date:   2017-02-15T03:51:58Z

Unify Fn API and Runner API coder spec




> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2011: [BEAM-115,BEAM-1348] Unify Fn API and Runner API co...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2011

[BEAM-115,BEAM-1348] Unify Fn API and Runner API coder specs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This includes the entirety of #2000, which unified `FunctionSpec`. Feel 
free to review that first or, if you prefer, review just this since the diff is 
small.

The unification here was uninteresting on top of #2000. Summary of changes:

 - Moved a coder's local id out of the `Coder` itself and into the key of a 
map in `ProcessBundleDescriptor`. Philosophically, the id is an essential 
aspect of `ProcessBundleDescriptor` (or `Pipeline`) but not not an essential 
aspect of a coder.  Pragmatically, this allows the Runner API and the Fn API to 
key the map on different types (`string` and `int64` respectively). 
Prospectively, it makes it easy to construct instances of the message that are 
"just values" without any id, which is aesthetically pleasing and more flexible 
to more uses.
 - Inlined the `SdkFunctionSpec` in `Coder` in the Runner API. Having it by 
reference introduces a needless sharing of key type and adds needless overhead, 
since coders are already stored by reference, as are environments.

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam fn-api-coders

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2011.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2011


commit 2b55a7f303ab0fea58ad279dd214253b4fe69565
Author: Kenneth Knowles 
Date:   2017-02-14T20:33:43Z

Remove underscore from Runner API proto Java package

commit 4e7865b828eae962532f1759833eed8b0e769cc9
Author: Kenneth Knowles 
Date:   2017-02-13T16:38:40Z

Unify Fn API and Runner API FunctionSpec

commit 5b5e6290e893385c47799cf5523c29be64c102fd
Author: Kenneth Knowles 
Date:   2017-02-15T03:51:58Z

Unify Fn API and Runner API coder spec




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #154: Clarify state further in capability matrix

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam-site/pull/154

Clarify state further in capability matrix

Some feedback after the blog post was that the term "Keyed State" in the 
capability matrix is lacking a referent. Basically I got more queries like 
[this 
one](https://lists.apache.org/thread.html/7ab688401796e1d7411c9975581ea5001a8c67fa415fff9f720753c9@%3Cdev.beam.apache.org%3E)
 but actually in response to a whole blog post describing it!

I had thought this term came from Millwheel so there might be some use in 
tying the two together. But it is actually [not in the 
paper](https://www.cs.cmu.edu/~pavlo/courses/fall2013/static/papers/p734-akidau.pdf).
 I think it is used a bit in Flink. A more general terminology seems 
appropriate for the capability matrix here.

In this PR I have suggested a general terminology, but there may be better 
options.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam-site state-capabilities

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/154.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #154


commit 50b61f8132cdf847255746df8a480c3c0224ff8a
Author: Kenneth Knowles 
Date:   2017-02-15T03:14:32Z

Clarify state further in capability matrix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #2295

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-1489) The DirectRunner does not discard existing PaneInfo during Window Assignment

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-1489:
--
Summary: The DirectRunner does not discard existing PaneInfo during Window 
Assignment  (was: The DirectRunner should not discard existing PaneInfo during 
Window Assignment)

> The DirectRunner does not discard existing PaneInfo during Window Assignment
> 
>
> Key: BEAM-1489
> URL: https://issues.apache.org/jira/browse/BEAM-1489
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
> Fix For: 0.6.0
>
>
> PaneInfo should be propagated when windows are reassigned, as they are in all 
> other non-firing transforms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1489) The DirectRunner should not discard existing PaneInfo during Window Assignment

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1489.
---
   Resolution: Fixed
Fix Version/s: 0.6.0

> The DirectRunner should not discard existing PaneInfo during Window Assignment
> --
>
> Key: BEAM-1489
> URL: https://issues.apache.org/jira/browse/BEAM-1489
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
> Fix For: 0.6.0
>
>
> PaneInfo should be propagated when windows are reassigned, as they are in all 
> other non-firing transforms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2294

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1489) The DirectRunner should not discard existing PaneInfo during Window Assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867075#comment-15867075
 ] 

ASF GitHub Bot commented on BEAM-1489:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2008


> The DirectRunner should not discard existing PaneInfo during Window Assignment
> --
>
> Key: BEAM-1489
> URL: https://issues.apache.org/jira/browse/BEAM-1489
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>
> PaneInfo should be propagated when windows are reassigned, as they are in all 
> other non-firing transforms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/3] beam git commit: This closes #2008

[tgroh]

This closes #2008


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9061c65e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9061c65e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9061c65e

Branch: refs/heads/master
Commit: 9061c65e3dbab4391a5534558e6af652d8b30945
Parents: 2685ba5 29fc84b
Author: Thomas Groh 
Authored: Tue Feb 14 17:31:35 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 14 17:31:35 2017 -0800

--
 .../beam/runners/core/WindowMatchers.java   |  3 +-
 .../runners/direct/WindowEvaluatorFactory.java  |  3 +-
 .../direct/WindowEvaluatorFactoryTest.java  | 41 ++--
 3 files changed, 24 insertions(+), 23 deletions(-)
--

[GitHub] beam pull request #2008: [BEAM-1489] Propagate Pane in WindowEvaluatorFactor...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2008


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam git commit: Propagate Pane in WindowEvaluatorFactory

[tgroh]

Propagate Pane in WindowEvaluatorFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/29fc84b2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/29fc84b2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/29fc84b2

Branch: refs/heads/master
Commit: 29fc84b2acd64b0028f189ff5350ed91e33854ad
Parents: 4e69a79
Author: Thomas Groh 
Authored: Tue Feb 14 16:19:54 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 14 17:31:34 2017 -0800

--
 .../runners/direct/WindowEvaluatorFactory.java  |  3 +-
 .../direct/WindowEvaluatorFactoryTest.java  | 41 ++--
 2 files changed, 22 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/29fc84b2/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WindowEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WindowEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WindowEvaluatorFactory.java
index 4ca556b..3cf178c 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WindowEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WindowEvaluatorFactory.java
@@ -25,7 +25,6 @@ import 
org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.transforms.windowing.Window.Bound;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
@@ -91,7 +90,7 @@ class WindowEvaluatorFactory implements 
TransformEvaluatorFactory {
 Collection windows = assignWindows(windowFn, 
element);
 outputBundle.add(
 WindowedValue.of(
-element.getValue(), element.getTimestamp(), windows, 
PaneInfo.NO_FIRING));
+element.getValue(), element.getTimestamp(), windows, 
element.getPane()));
   }
 }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/29fc84b2/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
index aa841ed..7e6eb2f 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
@@ -19,6 +19,7 @@ package org.apache.beam.runners.direct;
 
 import static 
org.apache.beam.runners.core.WindowMatchers.isSingleWindowedValue;
 import static org.apache.beam.runners.core.WindowMatchers.isWindowedValue;
+import static org.apache.beam.sdk.transforms.windowing.PaneInfo.NO_FIRING;
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.junit.Assert.assertThat;
 import static org.mockito.Mockito.when;
@@ -43,6 +44,7 @@ import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.transforms.windowing.NonMergingWindowFn;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
 import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.transforms.windowing.Window.Bound;
@@ -77,12 +79,12 @@ public class WindowEvaluatorFactoryTest {
   private WindowedValue valueInGlobalWindow =
   WindowedValue.timestampedValueInGlobalWindow(3L, new Instant(2L));
 
+  private final PaneInfo intervalWindowPane = PaneInfo.createPane(false, 
false, Timing.LATE, 3, 2);
   private WindowedValue valueInIntervalWindow =
   WindowedValue.of(
   Long.valueOf(2L),
   new Instant(-10L),
-  new IntervalWindow(new Instant(-100), EPOCH),
-  PaneInfo.NO_FIRING);
+  new IntervalWindow(new Instant(-100), EPOCH), intervalWindowPane);
 
   private IntervalWindow intervalWindow1 =
   new IntervalWindow(EPOCH, BoundedWindow.TIMESTAMP_MAX_VALUE);
@@ -91,12 +93,13 @@ public class WindowEvaluatorFactoryTest {
   new IntervalWindow(
   EPOCH.plus(Duration.standardDays(3)), 
EPOCH.plus(Duration.standardDays(6)));
 
+  private final PaneInfo multiWindowPane = PaneInfo.createPane

[1/3] beam git commit: Match Pane in WindowedValueMatcher

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 2685ba528 -> 9061c65e3


Match Pane in WindowedValueMatcher


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4e69a797
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4e69a797
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4e69a797

Branch: refs/heads/master
Commit: 4e69a7976faf32863eb796bfebf470332fcbd709
Parents: 2685ba5
Author: Thomas Groh 
Authored: Tue Feb 14 16:32:08 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 14 17:31:33 2017 -0800

--
 .../test/java/org/apache/beam/runners/core/WindowMatchers.java| 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4e69a797/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchers.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchers.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchers.java
index 6c3a7e2..9769d10 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchers.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/WindowMatchers.java
@@ -198,7 +198,8 @@ public class WindowMatchers {
 protected boolean matchesSafely(WindowedValue windowedValue) {
   return valueMatcher.matches(windowedValue.getValue())
   && timestampMatcher.matches(windowedValue.getTimestamp())
-  && windowsMatcher.matches(windowedValue.getWindows());
+  && windowsMatcher.matches(windowedValue.getWindows())
+  && paneInfoMatcher.matches(windowedValue.getPane());
 }
   }
 }

[jira] [Commented] (BEAM-1185) Remove the word Pipeline from the name of all PipelineRunner implementations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867047#comment-15867047
 ] 

ASF GitHub Bot commented on BEAM-1185:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2010

[BEAM-1185] Make examples blocking as command line invoked

R: @aaltay PTAL

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1185-make-examples-blocking

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2010.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2010


commit 5801ab49de1976265b07839792df84f1f21f68eb
Author: Sourabh Bajaj 
Date:   2017-02-15T01:15:04Z

Make examples blocking as command line invoked




> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-1185
> URL: https://issues.apache.org/jira/browse/BEAM-1185
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>  Labels: backward-incompatible, sdk-consistency
> Fix For: Not applicable
>
>
> See: https://issues.apache.org/jira/browse/BEAM-234
> Rename all runners to remove the Pipeline word from their name in the Python 
> SDK (e.g. DirectPipelineRunner -> DirectRunner).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2010: [BEAM-1185] Make examples blocking as command line ...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2010

[BEAM-1185] Make examples blocking as command line invoked

R: @aaltay PTAL

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1185-make-examples-blocking

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2010.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2010


commit 5801ab49de1976265b07839792df84f1f21f68eb
Author: Sourabh Bajaj 
Date:   2017-02-15T01:15:04Z

Make examples blocking as command line invoked




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1473) Remove unused windmill proto files from python sdk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867034#comment-15867034
 ] 

ASF GitHub Bot commented on BEAM-1473:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2001


> Remove unused windmill proto files from python sdk
> --
>
> Key: BEAM-1473
> URL: https://issues.apache.org/jira/browse/BEAM-1473
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/internal/windmill_pb2.py
>  
> There are two unused windmill files in beam that should be cleaned.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2001: [BEAM-1473] Remove unused windmill proto from pytho...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2001


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/4] beam git commit: [BEAM-1473] Remove unused windmill proto from python sdk

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 0d3389a20 -> 2685ba528


http://git-wip-us.apache.org/repos/asf/beam/blob/88b2d9b2/sdks/python/apache_beam/internal/windmill_service_pb2.py
--
diff --git a/sdks/python/apache_beam/internal/windmill_service_pb2.py 
b/sdks/python/apache_beam/internal/windmill_service_pb2.py
deleted file mode 100644
index b26fb63..000
--- a/sdks/python/apache_beam/internal/windmill_service_pb2.py
+++ /dev/null
@@ -1,164 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-# source: windmill_service.proto
-
-import sys
-_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
-from google.protobuf import descriptor as _descriptor
-from google.protobuf import message as _message
-from google.protobuf import reflection as _reflection
-from google.protobuf import symbol_database as _symbol_database
-from google.protobuf import descriptor_pb2
-# @@protoc_insertion_point(imports)
-
-_sym_db = _symbol_database.Default()
-
-
-import windmill_pb2 as windmill__pb2
-
-
-DESCRIPTOR = _descriptor.FileDescriptor(
-  name='windmill_service.proto',
-  package='google.dataflow.windmillservice.v1alpha1',
-  syntax='proto2',
-  
serialized_pb=_b('\n\x16windmill_service.proto\x12(google.dataflow.windmillservice.v1alpha1\x1a\x0ewindmill.proto2\xf9\x02\n\x1c\x43loudWindmillServiceV1Alpha1\x12>\n\x07GetWork\x12\x18.windmill.GetWorkRequest\x1a\x19.windmill.GetWorkResponse\x12>\n\x07GetData\x12\x18.windmill.GetDataRequest\x1a\x19.windmill.GetDataResponse\x12G\n\nCommitWork\x12\x1b.windmill.CommitWorkRequest\x1a\x1c.windmill.CommitWorkResponse\x12\x44\n\tGetConfig\x12\x1a.windmill.GetConfigRequest\x1a\x1b.windmill.GetConfigResponse\x12J\n\x0bReportStats\x12\x1c.windmill.ReportStatsRequest\x1a\x1d.windmill.ReportStatsResponseB7\n5com.apache_beam.sdk.runners.worker.windmill')
-  ,
-  dependencies=[windmill__pb2.DESCRIPTOR,])
-_sym_db.RegisterFileDescriptor(DESCRIPTOR)
-
-
-
-
-
-DESCRIPTOR.has_options = True
-DESCRIPTOR._options = _descriptor._ParseOptions(descriptor_pb2.FileOptions(), 
_b('\n5com.apache_beam.sdk.runners.worker.windmill'))
-from grpc.beta import implementations as beta_implementations
-from grpc.beta import interfaces as beta_interfaces
-from grpc.framework.common import cardinality
-from grpc.framework.interfaces.face import utilities as face_utilities
-
-
-class BetaCloudWindmillServiceV1Alpha1Servicer(object):
-  """The Cloud Windmill Service API used by GCE to acquire and process 
streaming
-  Dataflow work.
-  """
-  def GetWork(self, request, context):
-"""Gets streaming Dataflow work.
-"""
-context.code(beta_interfaces.StatusCode.UNIMPLEMENTED)
-  def GetData(self, request, context):
-"""Gets data from Windmill.
-"""
-context.code(beta_interfaces.StatusCode.UNIMPLEMENTED)
-  def CommitWork(self, request, context):
-"""Commits previously acquired work.
-"""
-context.code(beta_interfaces.StatusCode.UNIMPLEMENTED)
-  def GetConfig(self, request, context):
-"""Gets dependant configuration from windmill.
-"""
-context.code(beta_interfaces.StatusCode.UNIMPLEMENTED)
-  def ReportStats(self, request, context):
-"""Reports stats to Windmill.
-"""
-context.code(beta_interfaces.StatusCode.UNIMPLEMENTED)
-
-
-class BetaCloudWindmillServiceV1Alpha1Stub(object):
-  """The Cloud Windmill Service API used by GCE to acquire and process 
streaming
-  Dataflow work.
-  """
-  def GetWork(self, request, timeout, metadata=None, with_call=False, 
protocol_options=None):
-"""Gets streaming Dataflow work.
-"""
-raise NotImplementedError()
-  GetWork.future = None
-  def GetData(self, request, timeout, metadata=None, with_call=False, 
protocol_options=None):
-"""Gets data from Windmill.
-"""
-raise NotImplementedError()
-  GetData.future = None
-  def CommitWork(self, request, timeout, metadata=None, with_call=False, 
protocol_options=None):
-"""Commits previously acquired work.
-"""
-raise NotImplementedError()
-  CommitWork.future = None
-  def GetConfig(self, request, timeout, metadata=None, with_call=False, 
pro

[2/4] beam git commit: [BEAM-1473] Remove unused windmill proto from python sdk

[altay]

http://git-wip-us.apache.org/repos/asf/beam/blob/88b2d9b2/sdks/python/apache_beam/internal/windmill_pb2.py
--
diff --git a/sdks/python/apache_beam/internal/windmill_pb2.py 
b/sdks/python/apache_beam/internal/windmill_pb2.py
deleted file mode 100644
index c9a5c4e..000
--- a/sdks/python/apache_beam/internal/windmill_pb2.py
+++ /dev/null
@@ -1,2278 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-# source: windmill.proto
-
-import sys
-_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
-from google.protobuf import descriptor as _descriptor
-from google.protobuf import message as _message
-from google.protobuf import reflection as _reflection
-from google.protobuf import symbol_database as _symbol_database
-from google.protobuf import descriptor_pb2
-# @@protoc_insertion_point(imports)
-
-_sym_db = _symbol_database.Default()
-
-
-
-
-DESCRIPTOR = _descriptor.FileDescriptor(
-  name='windmill.proto',
-  package='windmill',
-  syntax='proto2',
-  
serialized_pb=_b('\n\x0ewindmill.proto\x12\x08windmill\"R\n\x07Message\x12\'\n\ttimestamp\x18\x01
 \x02(\x03:\x14-9223372036854775808\x12\x0c\n\x04\x64\x61ta\x18\x02 
\x02(\x0c\x12\x10\n\x08metadata\x18\x03 
\x01(\x0c\"\xbf\x01\n\x05Timer\x12\x0b\n\x03tag\x18\x01 
\x02(\x0c\x12\'\n\ttimestamp\x18\x02 
\x01(\x03:\x14-9223372036854775808\x12-\n\x04type\x18\x03 
\x01(\x0e\x32\x14.windmill.Timer.Type:\tWATERMARK\x12\x14\n\x0cstate_family\x18\x04
 
\x01(\t\";\n\x04Type\x12\r\n\tWATERMARK\x10\x00\x12\x0c\n\x08REALTIME\x10\x01\x12\x16\n\x12\x44\x45PENDENT_REALTIME\x10\x02\"X\n\x12InputMessageBundle\x12\x1d\n\x15source_computation_id\x18\x01
 \x02(\t\x12#\n\x08messages\x18\x02 
\x03(\x0b\x32\x11.windmill.Message\"r\n\x12KeyedMessageBundle\x12\x0b\n\x03key\x18\x01
 \x02(\x0c\x12\x14\n\x0csharding_key\x18\x04 
\x01(\x06\x12#\n\x08messages\x18\x02 
\x03(\x0b\x32\x11.windmill.Message\x12\x14\n\x0cmessages_ids\x18\x03 
\x03(\x0c\"\x87\x01\n\x13OutputMessageBundle\x12\"\n\x1a\x64\x65stination_computation_id\
 x18\x01 \x01(\t\x12\x1d\n\x15\x64\x65stination_stream_id\x18\x03 
\x01(\t\x12-\n\x07\x62undles\x18\x02 
\x03(\x0b\x32\x1c.windmill.KeyedMessageBundle\"t\n\x13PubSubMessageBundle\x12\r\n\x05topic\x18\x01
 \x02(\t\x12#\n\x08messages\x18\x02 
\x03(\x0b\x32\x11.windmill.Message\x12\x17\n\x0ftimestamp_label\x18\x03 
\x01(\t\x12\x10\n\x08id_label\x18\x04 
\x01(\t\".\n\x0bTimerBundle\x12\x1f\n\x06timers\x18\x01 
\x03(\x0b\x32\x0f.windmill.Timer\">\n\x05Value\x12\'\n\ttimestamp\x18\x01 
\x02(\x03:\x14-9223372036854775808\x12\x0c\n\x04\x64\x61ta\x18\x02 
\x02(\x0c\"M\n\x08TagValue\x12\x0b\n\x03tag\x18\x01 
\x02(\x0c\x12\x1e\n\x05value\x18\x02 
\x01(\x0b\x32\x0f.windmill.Value\x12\x14\n\x0cstate_family\x18\x03 
\x01(\t\"\xdb\x01\n\x07TagList\x12\x0b\n\x03tag\x18\x01 
\x02(\x0c\x12+\n\rend_timestamp\x18\x02 
\x01(\x03:\x14-9223372036854775808\x12\x1f\n\x06values\x18\x03 
\x03(\x0b\x32\x0f.windmill.Value\x12\x14\n\x0cstate_family\x18\x04 
\x01(\t\x12\x15\n\rrequest_token\x18\x07 \x01(\x0c\x12\x1a\n\x12\x63onti
 nuation_token\x18\x05 \x01(\x0c\x12,\n\x0f\x66\x65tch_max_bytes\x18\x06 
\x01(\x03:\x13\x39\x32\x32\x33\x33\x37\x32\x30\x33\x36\x38\x35\x34\x37\x37\x35\x38\x30\x37\",\n\x0cGlobalDataId\x12\x0b\n\x03tag\x18\x01
 \x02(\t\x12\x0f\n\x07version\x18\x02 
\x02(\x0c\"k\n\nGlobalData\x12\'\n\x07\x64\x61ta_id\x18\x01 
\x02(\x0b\x32\x16.windmill.GlobalDataId\x12\x10\n\x08is_ready\x18\x02 
\x01(\x08\x12\x0c\n\x04\x64\x61ta\x18\x03 
\x01(\x0c\x12\x14\n\x0cstate_family\x18\x04 
\x01(\t\"I\n\x0bSourceState\x12\r\n\x05state\x18\x01 
\x01(\x0c\x12\x14\n\x0c\x66inalize_ids\x18\x02 
\x03(\x06\x12\x15\n\ronly_finalize\x18\x03 
\x01(\x08\"Y\n\rWatermarkHold\x12\x0b\n\x03tag\x18\x01 
\x02(\x0c\x12\x16\n\ntimestamps\x18\x02 
\x03(\x03\x42\x02\x10\x01\x12\r\n\x05reset\x18\x03 
\x01(\x08\x12\x14\n\x0cstate_family\x18\x04 
\x01(\t\"\xd4\x02\n\x08WorkItem\x12\x0b\n\x03key\x18\x01 
\x02(\x0c\x12\x12\n\nwork_token\x18\x02 
\x02(\x06\x12\x14\n\x0csharding_key\x18\t 
\x01(\x06\x12\x13\n\x0b\x63\x61\x63he_token\x18\x07 \x01(\x06\x
 12\x35\n\x0fmessage_bundles\x18\x03 
\x03(\x0b\x32\x1c.windmill.InputMessageBundle\

[4/4] beam git commit: This closes #2001

[altay]

This closes #2001


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2685ba52
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2685ba52
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2685ba52

Branch: refs/heads/master
Commit: 2685ba528d1c9c0dd1d4900dc8e84c9f78c0e5d8
Parents: 0d3389a 88b2d9b
Author: Ahmet Altay 
Authored: Tue Feb 14 16:16:12 2017 -0800
Committer: Ahmet Altay 
Committed: Tue Feb 14 16:16:12 2017 -0800

--
 sdks/python/.pylintrc   |2 +-
 .../python/apache_beam/internal/windmill_pb2.py | 2278 --
 .../internal/windmill_service_pb2.py|  164 --
 sdks/python/run_pylint.sh   |2 -
 4 files changed, 1 insertion(+), 2445 deletions(-)
--

[3/4] beam git commit: [BEAM-1473] Remove unused windmill proto from python sdk

[altay]

[BEAM-1473] Remove unused windmill proto from python sdk


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/88b2d9b2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/88b2d9b2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/88b2d9b2

Branch: refs/heads/master
Commit: 88b2d9b20034f6f561195707b2316ddb4dfd43de
Parents: 0d3389a
Author: Sourabh Bajaj 
Authored: Tue Feb 14 16:13:48 2017 -0800
Committer: Ahmet Altay 
Committed: Tue Feb 14 16:16:01 2017 -0800

--
 sdks/python/.pylintrc   |2 +-
 .../python/apache_beam/internal/windmill_pb2.py | 2278 --
 .../internal/windmill_service_pb2.py|  164 --
 sdks/python/run_pylint.sh   |2 -
 4 files changed, 1 insertion(+), 2445 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/88b2d9b2/sdks/python/.pylintrc
--
diff --git a/sdks/python/.pylintrc b/sdks/python/.pylintrc
index c69fd2b..7a0611a 100644
--- a/sdks/python/.pylintrc
+++ b/sdks/python/.pylintrc
@@ -17,7 +17,7 @@
 
 [MASTER]
 # Ignore auto-generated files.
-ignore=clients,windmill_pb2.py,windmill_service_pb2.py
+ignore=clients
 
 [BASIC]
 # Regular expression which should only match the name

[jira] [Updated] (BEAM-646) Get runners out of the apply()

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-646:
-
Labels: backwards-incompatible  (was: )

> Get runners out of the apply()
> --
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>  Labels: backwards-incompatible
>
> Right now, the runner intercepts calls to apply() and replaces transforms as 
> we go. This means that there is no "original" user graph. For portability and 
> misc architectural benefits, we would like to build the original graph first, 
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more 
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline 
> object.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-958) desiredNumWorkers in Dataflow is too low

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867015#comment-15867015
 ] 

ASF GitHub Bot commented on BEAM-958:
-

GitHub user rangadi opened a pull request:

https://github.com/apache/beam/pull/2009

[BEAM-958] Improve desired number of splits in Dataflow.

Set desired number of splits for unbounded sources to 4 x (max)workers.
I am not sure of the rationale for previous values, especially 
1xMaxWorkers, which implies there could be just core that is utilized on a 
worker.

This fix is important for supporting autoscaling with sources that respect 
'desiredNumSplits' passed to `generateInitialSplits()` (KafkaIO is one such 
source). 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rangadi/incubator-beam max_workers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2009.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2009


commit 628fe1c618a911f3de1f702124e5336ad2b4ac44
Author: Raghu Angadi 
Date:   2017-02-15T00:40:03Z

[BEAM-958] Improve desired number of splits in Dataflow.




> desiredNumWorkers in Dataflow is too low
> 
>
> Key: BEAM-958
> URL: https://issues.apache.org/jira/browse/BEAM-958
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>  Labels: backward-incompatible
>
> {{desiredNumWorkers}} in [UnboundedSource 
> API|https://github.com/apache/incubator-beam/blob/v0.3.0-incubating-RC1/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L69]
>  is a suggestion to a source about how many splits it should create. KafkaIO 
> currently takes this literally and only creates up to this many splits.
> The main draw back is that it is very low in Dataflow. It is calculated as 
>   * {{1 * maxNumWorkers}} if {{--maxNumWorkers}} is specified, otherwise
>   * {{3 * numWorkers}}.
> That implies there is only single reader per worker (which is usually a 4 
> core VM). That can leave CPU under utilized on many pipelines.
> Even 3x in case of fixes number of workers seems low to me. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2009: [BEAM-958] Improve desired number of splits in Data...

[rangadi]

GitHub user rangadi opened a pull request:

https://github.com/apache/beam/pull/2009

[BEAM-958] Improve desired number of splits in Dataflow.

Set desired number of splits for unbounded sources to 4 x (max)workers.
I am not sure of the rationale for previous values, especially 
1xMaxWorkers, which implies there could be just core that is utilized on a 
worker.

This fix is important for supporting autoscaling with sources that respect 
'desiredNumSplits' passed to `generateInitialSplits()` (KafkaIO is one such 
source). 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rangadi/incubator-beam max_workers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2009.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2009


commit 628fe1c618a911f3de1f702124e5336ad2b4ac44
Author: Raghu Angadi 
Date:   2017-02-15T00:40:03Z

[BEAM-958] Improve desired number of splits in Dataflow.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1488) add ConsoleIO

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1488:
---
Component/s: (was: sdk-java-core)
 sdk-java-extensions

> add ConsoleIO 
> --
>
> Key: BEAM-1488
> URL: https://issues.apache.org/jira/browse/BEAM-1488
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>
> Add a ConsoleIO, which can read from console as input, and output to console 
> as output. I use this a lot for local debugging/testing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1488) add ConsoleIO

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1488:
--

Assignee: (was: Davor Bonaci)

> add ConsoleIO 
> --
>
> Key: BEAM-1488
> URL: https://issues.apache.org/jira/browse/BEAM-1488
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>
> Add a ConsoleIO, which can read from console as input, and output to console 
> as output. I use this a lot for local debugging/testing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1489) The DirectRunner should not discard existing PaneInfo during Window Assignment

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1489:
-

 Summary: The DirectRunner should not discard existing PaneInfo 
during Window Assignment
 Key: BEAM-1489
 URL: https://issues.apache.org/jira/browse/BEAM-1489
 Project: Beam
  Issue Type: Bug
  Components: runner-direct
Reporter: Thomas Groh
Assignee: Thomas Groh
Priority: Minor


PaneInfo should be propagated when windows are reassigned, as they are in all 
other non-firing transforms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2008: [BEAM-1489] Propagate Pane in WindowEvaluatorFactor...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2008

[BEAM-1489] Propagate Pane in WindowEvaluatorFactory

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam fix_window_assignment

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2008.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2008


commit a76526da9f7f8b7d45305a1312e05718e9672f78
Author: Thomas Groh 
Date:   2017-02-15T00:19:54Z

Propagate Pane in WindowEvaluatorFactory




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1489) The DirectRunner should not discard existing PaneInfo during Window Assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866989#comment-15866989
 ] 

ASF GitHub Bot commented on BEAM-1489:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2008

[BEAM-1489] Propagate Pane in WindowEvaluatorFactory

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam fix_window_assignment

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2008.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2008


commit a76526da9f7f8b7d45305a1312e05718e9672f78
Author: Thomas Groh 
Date:   2017-02-15T00:19:54Z

Propagate Pane in WindowEvaluatorFactory




> The DirectRunner should not discard existing PaneInfo during Window Assignment
> --
>
> Key: BEAM-1489
> URL: https://issues.apache.org/jira/browse/BEAM-1489
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>
> PaneInfo should be propagated when windows are reassigned, as they are in all 
> other non-firing transforms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1113) Support for new Timer API in Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866881#comment-15866881
 ] 

ASF GitHub Bot commented on BEAM-1113:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2007

[BEAM-1113] Re-enable UsesTimersInParDo tests in Dataflow runner

A rather vital commit left out of #1792. Do not review yet; utilizing 
Jenkins.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DataflowRunner-timers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2007.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2007


commit cc231914754c98708159e05fd85c95aa193a0a46
Author: Kenneth Knowles 
Date:   2017-02-14T22:54:11Z

Re-enable UsesTimersInParDo tests in Dataflow runner




> Support for new Timer API in Dataflow runner
> 
>
> Key: BEAM-1113
> URL: https://issues.apache.org/jira/browse/BEAM-1113
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2007: [BEAM-1113] Re-enable UsesTimersInParDo tests in Da...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2007

[BEAM-1113] Re-enable UsesTimersInParDo tests in Dataflow runner

A rather vital commit left out of #1792. Do not review yet; utilizing 
Jenkins.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam DataflowRunner-timers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2007.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2007


commit cc231914754c98708159e05fd85c95aa193a0a46
Author: Kenneth Knowles 
Date:   2017-02-14T22:54:11Z

Re-enable UsesTimersInParDo tests in Dataflow runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1485) pipeline_test:test_memory_usage is flaky

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1485.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> pipeline_test:test_memory_usage is flaky
> 
>
> Key: BEAM-1485
> URL: https://issues.apache.org/jira/browse/BEAM-1485
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: Not applicable
>
>
> The test is flaky, because it is sensitive to minor memory variations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1485) pipeline_test:test_memory_usage is flaky

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866865#comment-15866865
 ] 

ASF GitHub Bot commented on BEAM-1485:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2003


> pipeline_test:test_memory_usage is flaky
> 
>
> Key: BEAM-1485
> URL: https://issues.apache.org/jira/browse/BEAM-1485
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>
> The test is flaky, because it is sensitive to minor memory variations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Add some slack to the memory threshold for DirectRunner oom checker.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 2596d46a1 -> 0d3389a20


Add some slack to the memory threshold for DirectRunner oom checker.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8da2773d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8da2773d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8da2773d

Branch: refs/heads/master
Commit: 8da2773d04aa3018e7c0593c9a20a671eeaa5ad0
Parents: 2596d46
Author: Ahmet Altay 
Authored: Tue Feb 14 10:21:46 2017 -0800
Committer: Ahmet Altay 
Committed: Tue Feb 14 14:52:23 2017 -0800

--
 sdks/python/apache_beam/pipeline_test.py | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8da2773d/sdks/python/apache_beam/pipeline_test.py
--
diff --git a/sdks/python/apache_beam/pipeline_test.py 
b/sdks/python/apache_beam/pipeline_test.py
index e02ebc3..6c2512f 100644
--- a/sdks/python/apache_beam/pipeline_test.py
+++ b/sdks/python/apache_beam/pipeline_test.py
@@ -235,6 +235,9 @@ class PipelineTest(unittest.TestCase):
 memory_threshold = (
 get_memory_usage_in_bytes() + (3 * len_elements * num_elements))
 
+# Plus small additional slack for memory fluctuations during the test.
+memory_threshold += 10 * (2 ** 20)
+
 biglist = pipeline | 'oom:create' >> Create(
 ['x' * len_elements] * num_elements)
 for i in range(num_maps):

[2/2] beam git commit: This closes #2003

[altay]

This closes #2003


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0d3389a2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0d3389a2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0d3389a2

Branch: refs/heads/master
Commit: 0d3389a20ee44925dae64e9b7a60992bed5a6dd9
Parents: 2596d46 8da2773
Author: Ahmet Altay 
Authored: Tue Feb 14 14:52:27 2017 -0800
Committer: Ahmet Altay 
Committed: Tue Feb 14 14:52:27 2017 -0800

--
 sdks/python/apache_beam/pipeline_test.py | 3 +++
 1 file changed, 3 insertions(+)
--

[GitHub] beam pull request #2003: [BEAM-1485] Add some slack to the memory threshold ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2003


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-41) State for "Set" data, supporting efficient membership checks

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-41?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-41.
-
   Resolution: Fixed
Fix Version/s: 0.6.0

The API is in and runs on the direct runner. Separate tickets have been filed 
for support in each runner. I will update the capability matrix as well. Great!

> State for "Set" data, supporting efficient membership checks
> 
>
> Key: BEAM-41
> URL: https://issues.apache.org/jira/browse/BEAM-41
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
>Priority: Minor
>  Labels: State
> Fix For: 0.6.0
>
>
> Analogous to how BagState supports efficient blind writes, a SetState would 
> support efficient membership checking without reading the entire set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-42) State for "Map" data, supporting efficient key lookup, etc.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-42.
-
   Resolution: Fixed
Fix Version/s: 0.6.0

The API is in and runs on the direct runner. Separate tickets have been filed 
for support in each runner. I will update the capability matrix as well. Great!

> State for "Map" data, supporting efficient key lookup, etc.
> ---
>
> Key: BEAM-42
> URL: https://issues.apache.org/jira/browse/BEAM-42
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
>Priority: Minor
>  Labels: State
> Fix For: 0.6.0
>
>
> Analogous to BagState's blind writes, and SetState's efficient membership 
> check, MapState would add efficient key lookup and mutation, and other 
> Map-related operations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1488) add ConsoleIO

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-1488:


 Summary: add ConsoleIO 
 Key: BEAM-1488
 URL: https://issues.apache.org/jira/browse/BEAM-1488
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-core
Reporter: Xu Mingmin
Assignee: Davor Bonaci


Add a ConsoleIO, which can read from console as input, and output to console as 
output. I use this a lot for local debugging/testing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1950: [BEAM-41] [BEAM-42] State for "Set" data and "Map" ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1950


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/5] beam git commit: Exclude UsesSetState and UsesMapState categories from Dataflow runner

[kenn]

Exclude UsesSetState and UsesMapState categories from Dataflow runner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/36604a01
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/36604a01
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/36604a01

Branch: refs/heads/master
Commit: 36604a01086e0a2bf25569f06c2dae343e1ba225
Parents: 674bead
Author: JingsongLi 
Authored: Tue Feb 14 10:16:07 2017 +0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 11:06:20 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/36604a01/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index d366ddc..0f75b74 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -80,6 +80,8 @@
   
 org.apache.beam.sdk.testing.UsesAttemptedMetrics,
 org.apache.beam.sdk.testing.UsesCommittedMetrics,
+org.apache.beam.sdk.testing.UsesSetState,
+org.apache.beam.sdk.testing.UsesMapState,
 org.apache.beam.sdk.testing.UsesTimersInParDo,
 org.apache.beam.sdk.testing.UsesSplittableParDo,
 org.apache.beam.sdk.testing.UsesUnboundedPCollections,

[jira] [Commented] (BEAM-646) Get runners out of the apply()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866785#comment-15866785
 ] 

ASF GitHub Bot commented on BEAM-646:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2006

[BEAM-646] Get the #apply out of the DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Use the Graph Surgery API in the DirectRunner.

The Forwarding View is currently required to ensure that the `WriteView` 
transform
is marked as the producer of the View proper. It is then replaced by the 
original view,
so the Pipeline writes the view to the correct location.

I'm going to revisit the producer-marking code soon, but this is safe to 
review.

I also missed invariant maintenance within TransformHierarchy within #1998,
so that's added here as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam surgery_ish

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2006.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2006


commit 46543d75a92b8c6c19b55581ec8253ffd76f4058
Author: Thomas Groh 
Date:   2017-02-09T19:37:33Z

Return a Forwarding View in View Overrides

This ensures that replacing an output will not fail to mark the
overridden view as produced by the replacement transform.

commit e950ba71d7cc6910c6be7dd39576bca3c96287de
Author: Thomas Groh 
Date:   2017-02-14T21:56:07Z

Add the replacement input to Unexpanded Inputs

commit a8bf815ea4f0ff3bc9701936cd1547d222eafebc
Author: Thomas Groh 
Date:   2017-02-09T19:45:06Z

Use Graph Surgery in the DirectRunner

Remove DirectRunner#apply(). This migrates the DirectRunner to work on a
runner-agnostic graph.




> Get runners out of the apply()
> --
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now, the runner intercepts calls to apply() and replaces transforms as 
> we go. This means that there is no "original" user graph. For portability and 
> misc architectural benefits, we would like to build the original graph first, 
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more 
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline 
> object.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2006: [BEAM-646] Get the #apply out of the DirectRunner

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2006

[BEAM-646] Get the #apply out of the DirectRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Use the Graph Surgery API in the DirectRunner.

The Forwarding View is currently required to ensure that the `WriteView` 
transform
is marked as the producer of the View proper. It is then replaced by the 
original view,
so the Pipeline writes the view to the correct location.

I'm going to revisit the producer-marking code soon, but this is safe to 
review.

I also missed invariant maintenance within TransformHierarchy within #1998,
so that's added here as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam surgery_ish

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2006.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2006


commit 46543d75a92b8c6c19b55581ec8253ffd76f4058
Author: Thomas Groh 
Date:   2017-02-09T19:37:33Z

Return a Forwarding View in View Overrides

This ensures that replacing an output will not fail to mark the
overridden view as produced by the replacement transform.

commit e950ba71d7cc6910c6be7dd39576bca3c96287de
Author: Thomas Groh 
Date:   2017-02-14T21:56:07Z

Add the replacement input to Unexpanded Inputs

commit a8bf815ea4f0ff3bc9701936cd1547d222eafebc
Author: Thomas Groh 
Date:   2017-02-09T19:45:06Z

Use Graph Surgery in the DirectRunner

Remove DirectRunner#apply(). This migrates the DirectRunner to work on a
runner-agnostic graph.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-41) State for "Set" data, supporting efficient membership checks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866787#comment-15866787
 ] 

ASF GitHub Bot commented on BEAM-41:


Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1950


> State for "Set" data, supporting efficient membership checks
> 
>
> Key: BEAM-41
> URL: https://issues.apache.org/jira/browse/BEAM-41
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
>Priority: Minor
>  Labels: State
>
> Analogous to how BagState supports efficient blind writes, a SetState would 
> support efficient membership checking without reading the entire set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/5] beam git commit: Add UsesMapState and UsesSetState JUnit categories

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master ed3ef11fa -> 2596d46a1


Add UsesMapState and UsesSetState JUnit categories


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/674bead8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/674bead8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/674bead8

Branch: refs/heads/master
Commit: 674bead8c346421ebfadc3e0b2e13eb79e0280aa
Parents: bea101a
Author: JingsongLi 
Authored: Tue Feb 14 14:52:05 2017 +0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 11:02:48 2017 -0800

--
 .../apache/beam/sdk/testing/UsesMapState.java   | 25 
 .../apache/beam/sdk/testing/UsesSetState.java   | 25 
 2 files changed, 50 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/674bead8/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesMapState.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesMapState.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesMapState.java
new file mode 100644
index 000..9bced41
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesMapState.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+import org.apache.beam.sdk.util.state.MapState;
+
+/**
+ * Category tag for validation tests which utilize {@link MapState}.
+ */
+public interface UsesMapState {}

http://git-wip-us.apache.org/repos/asf/beam/blob/674bead8/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSetState.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSetState.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSetState.java
new file mode 100644
index 000..6fd74bd
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSetState.java
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.testing;
+
+import org.apache.beam.sdk.util.state.SetState;
+
+/**
+ * Category tag for validation tests which utilize {@link SetState}.
+ */
+public interface UsesSetState {}

[4/5] beam git commit: Add SetState and MapState

[kenn]

Add SetState and MapState


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a0702f5b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a0702f5b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a0702f5b

Branch: refs/heads/master
Commit: a0702f5bed3c7269e90b4702266945aa34dd1aea
Parents: 0f48321
Author: JingsongLi 
Authored: Tue Feb 14 14:52:05 2017 +0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 11:06:29 2017 -0800

--
 .../translation/utils/ApexStateInternals.java   |  18 ++
 .../runners/core/InMemoryStateInternals.java| 205 ++
 .../apache/beam/runners/core/StateMerging.java  |  44 +++
 .../org/apache/beam/runners/core/StateTag.java  |   8 +
 .../org/apache/beam/runners/core/StateTags.java |  30 ++
 .../core/InMemoryStateInternalsTest.java| 280 +--
 .../apache/beam/runners/core/StateTagTest.java  |  33 +++
 .../CopyOnAccessInMemoryStateInternals.java |  46 +++
 .../CopyOnAccessInMemoryStateInternalsTest.java |  58 
 .../wrappers/streaming/FlinkStateInternals.java |  18 ++
 .../apache/beam/sdk/util/state/MapState.java|  93 ++
 .../apache/beam/sdk/util/state/SetState.java|  71 +
 .../apache/beam/sdk/util/state/StateBinder.java |   6 +
 .../apache/beam/sdk/util/state/StateSpecs.java  |  89 ++
 .../apache/beam/sdk/transforms/ParDoTest.java   |  94 +++
 15 files changed, 1063 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a0702f5b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternals.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternals.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternals.java
index 34d993f..7634366 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternals.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/utils/ApexStateInternals.java
@@ -45,7 +45,9 @@ import org.apache.beam.sdk.transforms.windowing.OutputTimeFn;
 import org.apache.beam.sdk.util.CombineFnUtil;
 import org.apache.beam.sdk.util.state.AccumulatorCombiningState;
 import org.apache.beam.sdk.util.state.BagState;
+import org.apache.beam.sdk.util.state.MapState;
 import org.apache.beam.sdk.util.state.ReadableState;
+import org.apache.beam.sdk.util.state.SetState;
 import org.apache.beam.sdk.util.state.State;
 import org.apache.beam.sdk.util.state.StateContext;
 import org.apache.beam.sdk.util.state.StateContexts;
@@ -121,6 +123,22 @@ public class ApexStateInternals implements 
StateInternals, Serializable {
 }
 
 @Override
+public  SetState bindSet(
+StateTag> address,
+Coder elemCoder) {
+  throw new UnsupportedOperationException(
+  String.format("%s is not supported", 
SetState.class.getSimpleName()));
+}
+
+@Override
+public  MapState bindMap(
+StateTag> spec,
+Coder mapKeyCoder, Coder mapValueCoder) {
+  throw new UnsupportedOperationException(
+  String.format("%s is not supported", 
MapState.class.getSimpleName()));
+}
+
+@Override
 public  AccumulatorCombiningState
 bindCombiningValue(
 StateTag> address,

http://git-wip-us.apache.org/repos/asf/beam/blob/a0702f5b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
index 6a181f3..b4b2b38 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
@@ -17,10 +17,16 @@
  */
 package org.apache.beam.runners.core;
 
+import static com.google.common.base.Preconditions.checkNotNull;
+
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
 import java.util.List;
+import java.util.Map;
 import java.util.Objects;
+import java.util.Set;
 import javax.annotation.Nullable;
 import org.apache.beam.runners.core.StateTag.StateBinder;
 import org.apache.beam.sdk.annotations.Experimental;
@@ -34,7 +40,9 @@ import org.apache.beam.sdk.transforms.windowing.OutputTimeFn;
 import org.apache.beam.sdk.util.CombineFnUtil;
 import org.apache.beam.sdk.util.state.AccumulatorCombiningState;
 import org.apache.beam.sdk.util.state.BagState;
+import org.apache.beam.sdk.util.state.MapState;
 impo

[3/5] beam git commit: Exclude UsesSetState and UsesMapState tests for Flink runner

[kenn]

Exclude UsesSetState and UsesMapState tests for Flink runner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0f48321e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0f48321e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0f48321e

Branch: refs/heads/master
Commit: 0f48321e3b933904fbeb3fbe7c2a06741ff18ff0
Parents: 36604a0
Author: JingsongLi 
Authored: Tue Feb 14 10:16:07 2017 +0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 11:06:28 2017 -0800

--
 runners/flink/runner/pom.xml | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0f48321e/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index c800ed3..f254d9a 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -88,6 +88,8 @@
   
org.apache.beam.sdk.testing.RunnableOnService
   
 org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders,
+org.apache.beam.sdk.testing.UsesSetState,
+org.apache.beam.sdk.testing.UsesMapState,
 org.apache.beam.sdk.testing.UsesTimersInParDo,
 org.apache.beam.sdk.testing.UsesSplittableParDo,
 org.apache.beam.sdk.testing.UsesAttemptedMetrics,

[5/5] beam git commit: This closes #1950: State for "Set" data and "Map" data

[kenn]

This closes #1950: State for "Set" data and "Map" data

  Add SetState and MapState
  Exclude UsesSetState and UsesMapState tests for Flink runner
  Exclude UsesSetState and UsesMapState categories from Dataflow runner
  Add UsesMapState and UsesSetState JUnit categories


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2596d46a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2596d46a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2596d46a

Branch: refs/heads/master
Commit: 2596d46a1ebbcfb72caa5b3a943ffe7d8d7f4ba2
Parents: ed3ef11 a0702f5
Author: Kenneth Knowles 
Authored: Tue Feb 14 13:57:38 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Feb 14 13:57:38 2017 -0800

--
 .../translation/utils/ApexStateInternals.java   |  18 ++
 .../runners/core/InMemoryStateInternals.java| 205 ++
 .../apache/beam/runners/core/StateMerging.java  |  44 +++
 .../org/apache/beam/runners/core/StateTag.java  |   8 +
 .../org/apache/beam/runners/core/StateTags.java |  30 ++
 .../core/InMemoryStateInternalsTest.java| 280 +--
 .../apache/beam/runners/core/StateTagTest.java  |  33 +++
 .../CopyOnAccessInMemoryStateInternals.java |  46 +++
 .../CopyOnAccessInMemoryStateInternalsTest.java |  58 
 runners/flink/runner/pom.xml|   2 +
 .../wrappers/streaming/FlinkStateInternals.java |  18 ++
 runners/google-cloud-dataflow-java/pom.xml  |   2 +
 .../apache/beam/sdk/testing/UsesMapState.java   |  25 ++
 .../apache/beam/sdk/testing/UsesSetState.java   |  25 ++
 .../apache/beam/sdk/util/state/MapState.java|  93 ++
 .../apache/beam/sdk/util/state/SetState.java|  71 +
 .../apache/beam/sdk/util/state/StateBinder.java |   6 +
 .../apache/beam/sdk/util/state/StateSpecs.java  |  89 ++
 .../apache/beam/sdk/transforms/ParDoTest.java   |  94 +++
 19 files changed, 1117 insertions(+), 30 deletions(-)
--

[1/2] beam git commit: This closes #2005

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 345c2acca -> ed3ef11fa


This closes #2005


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ed3ef11f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ed3ef11f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ed3ef11f

Branch: refs/heads/master
Commit: ed3ef11fa160047473969ffe2a730caecb92f3de
Parents: 345c2ac 13e0915
Author: Thomas Groh 
Authored: Tue Feb 14 13:54:04 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 14 13:54:04 2017 -0800

--
 sdks/java/harness/pom.xml | 12 
 1 file changed, 12 insertions(+)
--

[GitHub] beam pull request #2005: [BEAM-1487] Suppress flaky harness tests

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2005


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1487) BufferingStreamObserverTest, BeamFnLoggingClientTest repeatedly times out in precommit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866780#comment-15866780
 ] 

ASF GitHub Bot commented on BEAM-1487:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2005


> BufferingStreamObserverTest, BeamFnLoggingClientTest repeatedly times out in 
> precommit
> --
>
> Key: BEAM-1487
> URL: https://issues.apache.org/jira/browse/BEAM-1487
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-fn-api
>Reporter: Thomas Groh
>Assignee: Luke Cwik
>Priority: Blocker
>
> BufferingStreamObserverTest
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7401/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7248/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7267/
> BeamFnLoggingClientTest
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7274/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7256/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7265/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7288/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7254/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7111/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: Suppress two Flaky Harness Tests

[tgroh]

Suppress two Flaky Harness Tests

These tests are hanging forever in Jenkins


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/13e09151
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/13e09151
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/13e09151

Branch: refs/heads/master
Commit: 13e091516049f83f0d6ae8e24f3840269de15f19
Parents: 345c2ac
Author: Thomas Groh 
Authored: Tue Feb 14 12:48:34 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 14 13:54:04 2017 -0800

--
 sdks/java/harness/pom.xml | 12 
 1 file changed, 12 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/13e09151/sdks/java/harness/pom.xml
--
diff --git a/sdks/java/harness/pom.xml b/sdks/java/harness/pom.xml
index e164ee0..3abe70b 100644
--- a/sdks/java/harness/pom.xml
+++ b/sdks/java/harness/pom.xml
@@ -44,6 +44,18 @@
   1.8
 
   
+
+  
+org.apache.maven.plugins
+maven-surefire-plugin
+
+  
+
+
org.apache.beam.fn.harness.logging.BeamFnLoggingClientTest
+
org.apache.beam.fn.harness.stream.BufferingStreamObserverTest
+  
+
+  
 
   
 

[jira] [Commented] (BEAM-646) Get runners out of the apply()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866698#comment-15866698
 ] 

ASF GitHub Bot commented on BEAM-646:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1973


> Get runners out of the apply()
> --
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now, the runner intercepts calls to apply() and replaces transforms as 
> we go. This means that there is no "original" user graph. For portability and 
> misc architectural benefits, we would like to build the original graph first, 
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more 
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline 
> object.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-646) Get runners out of the apply()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866701#comment-15866701
 ] 

ASF GitHub Bot commented on BEAM-646:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1998


> Get runners out of the apply()
> --
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now, the runner intercepts calls to apply() and replaces transforms as 
> we go. This means that there is no "original" user graph. For portability and 
> misc architectural benefits, we would like to build the original graph first, 
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more 
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline 
> object.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1998: [BEAM-646] Add Pipeline#replaceTransform

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1998


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Add Pipeline#replaceTransform

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 938a66ec3 -> 345c2acca


Add Pipeline#replaceTransform

This is the base method for Pipeline Surgery. It takes a
PTransformMatcher and a PTransformOverrideFactory, and replaces all
matching PTransforms with the result of the Override Factory.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/16c051ac
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/16c051ac
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/16c051ac

Branch: refs/heads/master
Commit: 16c051aca7fadfe3935988488f29723b932e47c7
Parents: 938a66e
Author: Thomas Groh 
Authored: Thu Feb 9 11:40:50 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 14 13:12:56 2017 -0800

--
 .../main/java/org/apache/beam/sdk/Pipeline.java |  63 -
 .../beam/sdk/runners/TransformHierarchy.java| 157 +---
 .../apache/beam/sdk/values/TupleTagList.java|   6 +
 .../sdk/runners/TransformHierarchyTest.java | 237 ++-
 4 files changed, 431 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/16c051ac/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
index eb0b199..109424d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
@@ -27,9 +27,14 @@ import java.util.Set;
 import org.apache.beam.sdk.coders.CoderRegistry;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptions.CheckEnabled;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.runners.PTransformMatcher;
+import org.apache.beam.sdk.runners.PTransformOverrideFactory;
+import org.apache.beam.sdk.runners.PTransformOverrideFactory.ReplacementOutput;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.runners.TransformHierarchy;
+import org.apache.beam.sdk.runners.TransformHierarchy.Node;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.PTransform;
@@ -167,6 +172,33 @@ public class Pipeline {
 return begin().apply(name, root);
   }
 
+  public void replace(
+  final PTransformMatcher matcher, PTransformOverrideFactory 
replacementFactory) {
+final Collection matches = new ArrayList<>();
+transforms.visit(
+new PipelineVisitor.Defaults() {
+  @Override
+  public CompositeBehavior enterCompositeTransform(Node node) {
+if (!node.isRootNode() && 
matcher.matches(node.toAppliedPTransform())) {
+  matches.add(node);
+  // This node will be replaced. It should not be visited.
+  return CompositeBehavior.DO_NOT_ENTER_TRANSFORM;
+}
+return CompositeBehavior.ENTER_TRANSFORM;
+  }
+
+  @Override
+  public void visitPrimitiveTransform(Node node) {
+if (matcher.matches(node.toAppliedPTransform())) {
+  matches.add(node);
+}
+  }
+});
+for (Node match : matches) {
+  applyReplacement(match, replacementFactory);
+}
+  }
+
   /**
* Runs the {@link Pipeline} using its {@link PipelineRunner}.
*/
@@ -323,7 +355,7 @@ public class Pipeline {
 
   private final PipelineRunner runner;
   private final PipelineOptions options;
-  private final TransformHierarchy transforms = new TransformHierarchy();
+  private final TransformHierarchy transforms = new TransformHierarchy(this);
   private Collection values = new ArrayList<>();
   private Set usedFullNames = new HashSet<>();
   private CoderRegistry coderRegistry;
@@ -394,6 +426,35 @@ public class Pipeline {
 }
   }
 
+  private >
+  void applyReplacement(
+  Node original,
+  PTransformOverrideFactory 
replacementFactory) {
+// Names for top-level transforms have been assigned. Any new collisions 
are within a node
+// and its replacement.
+getOptions().setStableUniqueNames(CheckEnabled.OFF);
+PTransform replacement =
+replacementFactory.getReplacementTransform((TransformT) 
original.getTransform());
+if (replacement == original.getTransform()) {
+  return;
+}
+InputT originalInput = replacementFactory.getInput(original.getInputs(), 
this);
+
+LOG.debug("Replacing {} with {}", original, replacement);
+transforms.replaceNode(original, originalInput, replacement);
+try {
+  OutputT newOutput = runner.apply(replacement, originalInput);

[2/2] beam git commit: This closes #1998

[tgroh]

This closes #1998


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/345c2acc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/345c2acc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/345c2acc

Branch: refs/heads/master
Commit: 345c2acca9b502fa40209de76d845118282d58c7
Parents: 938a66e 16c051a
Author: Thomas Groh 
Authored: Tue Feb 14 13:12:57 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 14 13:12:57 2017 -0800

--
 .../main/java/org/apache/beam/sdk/Pipeline.java |  63 -
 .../beam/sdk/runners/TransformHierarchy.java| 157 +---
 .../apache/beam/sdk/values/TupleTagList.java|   6 +
 .../sdk/runners/TransformHierarchyTest.java | 237 ++-
 4 files changed, 431 insertions(+), 32 deletions(-)
--

[1/2] beam git commit: This closes #1973

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master bea101a44 -> 938a66ec3


This closes #1973


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/938a66ec
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/938a66ec
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/938a66ec

Branch: refs/heads/master
Commit: 938a66ec3bc6d78aa7cd1992038f68f90cc283af
Parents: bea101a 4bd6d83
Author: Thomas Groh 
Authored: Tue Feb 14 13:12:15 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 14 13:12:15 2017 -0800

--
 .../beam/runners/core/PTransformMatchers.java   |  83 +++
 .../runners/core/PTransformMatchersTest.java| 224 ---
 2 files changed, 276 insertions(+), 31 deletions(-)
--

[GitHub] beam pull request #1973: [BEAM-646] Add ParDo Matchers to PTransformMatchers

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1973


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Add ParDo Matchers to PTransformMatchers

[tgroh]

Add ParDo Matchers to PTransformMatchers

These match Splittable ParDos and ParDos that use State and Timers.

Update tests to remove excess generic args.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4bd6d833
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4bd6d833
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4bd6d833

Branch: refs/heads/master
Commit: 4bd6d833d11fdf7f3ccb7ad8d97735ea2fddfc07
Parents: bea101a
Author: Thomas Groh 
Authored: Thu Feb 9 11:43:24 2017 -0800
Committer: Thomas Groh 
Committed: Tue Feb 14 13:12:15 2017 -0800

--
 .../beam/runners/core/PTransformMatchers.java   |  83 +++
 .../runners/core/PTransformMatchersTest.java| 224 ---
 2 files changed, 276 insertions(+), 31 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4bd6d833/runners/core-java/src/main/java/org/apache/beam/runners/core/PTransformMatchers.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PTransformMatchers.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PTransformMatchers.java
index 362e8dc..1d7e24e 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/PTransformMatchers.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/PTransformMatchers.java
@@ -21,7 +21,12 @@ import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
 import org.apache.beam.sdk.runners.PTransformMatcher;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
+import 
org.apache.beam.sdk.transforms.reflect.DoFnSignature.ProcessElementMethod;
+import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
 
 /**
  * A {@link PTransformMatcher} that matches {@link PTransform PTransforms} 
based on the class of the
@@ -56,4 +61,82 @@ public class PTransformMatchers {
   return application.getTransform().getClass().equals(clazz);
 }
   }
+
+  /**
+   * A {@link PTransformMatcher} that matches a {@link ParDo.Bound} containing 
a {@link DoFn} that
+   * is splittable, as signified by {@link 
ProcessElementMethod#isSplittable()}.
+   */
+  public static PTransformMatcher splittableParDoSingle() {
+return new PTransformMatcher() {
+  @Override
+  public boolean matches(AppliedPTransform application) {
+PTransform transform = application.getTransform();
+if (transform instanceof ParDo.Bound) {
+  DoFn fn = ((ParDo.Bound) transform).getFn();
+  DoFnSignature signature = DoFnSignatures.signatureForDoFn(fn);
+  return signature.processElement().isSplittable();
+}
+return false;
+  }
+};
+  }
+
+  /**
+   * A {@link PTransformMatcher} that matches a {@link ParDo.Bound} containing 
a {@link DoFn} that
+   * uses state or timers, as specified by {@link DoFnSignature#usesState()} 
and
+   * {@link DoFnSignature#usesTimers()}.
+   */
+  public static PTransformMatcher stateOrTimerParDoSingle() {
+return new PTransformMatcher() {
+  @Override
+  public boolean matches(AppliedPTransform application) {
+PTransform transform = application.getTransform();
+if (transform instanceof ParDo.Bound) {
+  DoFn fn = ((ParDo.Bound) transform).getFn();
+  DoFnSignature signature = DoFnSignatures.signatureForDoFn(fn);
+  return signature.usesState() || signature.usesTimers();
+}
+return false;
+  }
+};
+  }
+
+  /**
+   * A {@link PTransformMatcher} that matches a {@link ParDo.BoundMulti} 
containing a {@link DoFn}
+   * that is splittable, as signified by {@link 
ProcessElementMethod#isSplittable()}.
+   */
+  public static PTransformMatcher splittableParDoMulti() {
+return new PTransformMatcher() {
+  @Override
+  public boolean matches(AppliedPTransform application) {
+PTransform transform = application.getTransform();
+if (transform instanceof ParDo.BoundMulti) {
+  DoFn fn = ((ParDo.BoundMulti) transform).getFn();
+  DoFnSignature signature = DoFnSignatures.signatureForDoFn(fn);
+  return signature.processElement().isSplittable();
+}
+return false;
+  }
+};
+  }
+
+  /**
+   * A {@link PTransformMatcher} that matches a {@link ParDo.BoundMulti} 
containing a {@link DoFn}
+   * that uses state or timers, as specified by {@link 
DoFnSignature#usesState()} and
+   * {@link DoFnSignature#usesTimers()}.
+   */
+  public static PTransformMatcher stateOrTimerParDoMulti() {
+

[jira] [Commented] (BEAM-1485) pipeline_test:test_memory_usage is flaky

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866680#comment-15866680
 ] 

ASF GitHub Bot commented on BEAM-1485:
--

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/2003


> pipeline_test:test_memory_usage is flaky
> 
>
> Key: BEAM-1485
> URL: https://issues.apache.org/jira/browse/BEAM-1485
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>
> The test is flaky, because it is sensitive to minor memory variations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1485) pipeline_test:test_memory_usage is flaky

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866681#comment-15866681
 ] 

ASF GitHub Bot commented on BEAM-1485:
--

GitHub user aaltay reopened a pull request:

https://github.com/apache/beam/pull/2003

[BEAM-1485] Add some slack to the memory threshold for DirectRunner oom 
checker

R: @chamikaramj 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam flake2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2003


commit c7e67d0361bbc81d5d17ebe0d276724013cd5e17
Author: Ahmet Altay 
Date:   2017-02-14T18:21:46Z

Add some slack to the memory threshold for DirectRunner oom checker.




> pipeline_test:test_memory_usage is flaky
> 
>
> Key: BEAM-1485
> URL: https://issues.apache.org/jira/browse/BEAM-1485
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>
> The test is flaky, because it is sensitive to minor memory variations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2003: [BEAM-1485] Add some slack to the memory threshold ...

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/2003


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2003: [BEAM-1485] Add some slack to the memory threshold ...

[aaltay]

GitHub user aaltay reopened a pull request:

https://github.com/apache/beam/pull/2003

[BEAM-1485] Add some slack to the memory threshold for DirectRunner oom 
checker

R: @chamikaramj 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam flake2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2003


commit c7e67d0361bbc81d5d17ebe0d276724013cd5e17
Author: Ahmet Altay 
Date:   2017-02-14T18:21:46Z

Add some slack to the memory threshold for DirectRunner oom checker.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1487) BufferingStreamObserverTest, BeamFnLoggingClientTest repeatedly times out in precommit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866644#comment-15866644
 ] 

ASF GitHub Bot commented on BEAM-1487:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2005

[BEAM-1487] Suppress flaky harness tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam suppress_flaky_harness_tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2005.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2005


commit e5aaa6e7c730cb9566e15277ff659e06ac89ba67
Author: Thomas Groh 
Date:   2017-02-09T19:37:33Z

Return a Forwarding View in View Overrides

This ensures that replacing an output will not fail to mark the
overridden view as produced by the replacement transform.

commit d03b553e90559c59d12d3df092c255df8caf8fd8
Author: Thomas Groh 
Date:   2017-02-09T19:40:50Z

Add Pipeline#replaceTransform

This is the base method for Pipeline Surgery. It takes a
PTransformMatcher and a PTransformOverrideFactory, and replaces all
matching PTransforms with the result of the Override Factory.

commit 9597db4c678f64b9fe9d5dadb9454b6159c120da
Author: Thomas Groh 
Date:   2017-02-09T19:43:24Z

Add ParDo Matchers to PTransformMatchers

These match Splittable ParDos and ParDos that use State and Timers.

commit f59334858a7f0df9e645bac73637b6c7210338cf
Author: Thomas Groh 
Date:   2017-02-09T19:45:06Z

Use Graph Surgery in the DirectRunner

Remove DirectRunner#apply(). This migrates the DirectRunner to work on a
runner-agnostic graph.

commit e8d9641807f473c6214e9a9bbb6a9280d898bd6c
Author: Thomas Groh 
Date:   2017-02-14T20:48:34Z

Suppress two Flaky Harness Tests

These tests are hanging forever in Jenkins




> BufferingStreamObserverTest, BeamFnLoggingClientTest repeatedly times out in 
> precommit
> --
>
> Key: BEAM-1487
> URL: https://issues.apache.org/jira/browse/BEAM-1487
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-fn-api
>Reporter: Thomas Groh
>Assignee: Luke Cwik
>Priority: Blocker
>
> BufferingStreamObserverTest
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7401/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7248/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7267/
> BeamFnLoggingClientTest
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7274/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7256/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7265/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7288/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7254/
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7111/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2005: [BEAM-1487] Suppress flaky harness tests

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2005

[BEAM-1487] Suppress flaky harness tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam suppress_flaky_harness_tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2005.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2005


commit e5aaa6e7c730cb9566e15277ff659e06ac89ba67
Author: Thomas Groh 
Date:   2017-02-09T19:37:33Z

Return a Forwarding View in View Overrides

This ensures that replacing an output will not fail to mark the
overridden view as produced by the replacement transform.

commit d03b553e90559c59d12d3df092c255df8caf8fd8
Author: Thomas Groh 
Date:   2017-02-09T19:40:50Z

Add Pipeline#replaceTransform

This is the base method for Pipeline Surgery. It takes a
PTransformMatcher and a PTransformOverrideFactory, and replaces all
matching PTransforms with the result of the Override Factory.

commit 9597db4c678f64b9fe9d5dadb9454b6159c120da
Author: Thomas Groh 
Date:   2017-02-09T19:43:24Z

Add ParDo Matchers to PTransformMatchers

These match Splittable ParDos and ParDos that use State and Timers.

commit f59334858a7f0df9e645bac73637b6c7210338cf
Author: Thomas Groh 
Date:   2017-02-09T19:45:06Z

Use Graph Surgery in the DirectRunner

Remove DirectRunner#apply(). This migrates the DirectRunner to work on a
runner-agnostic graph.

commit e8d9641807f473c6214e9a9bbb6a9280d898bd6c
Author: Thomas Groh 
Date:   2017-02-14T20:48:34Z

Suppress two Flaky Harness Tests

These tests are hanging forever in Jenkins




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1487) BufferingStreamObserverTest, BeamFnLoggingClientTest repeatedly times out in precommit

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1487:
-

 Summary: BufferingStreamObserverTest, BeamFnLoggingClientTest 
repeatedly times out in precommit
 Key: BEAM-1487
 URL: https://issues.apache.org/jira/browse/BEAM-1487
 Project: Beam
  Issue Type: Bug
  Components: beam-model-fn-api
Reporter: Thomas Groh
Assignee: Luke Cwik
Priority: Blocker


BufferingStreamObserverTest
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7401/
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7248/
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7267/

BeamFnLoggingClientTest
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7274/
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7256/
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7265/
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7288/
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7254/
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7111/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1486) Inline AfterDelayFromFirstElementStateMachine to AfterProcessingTimeStateMachine and simplify

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1486:
-

 Summary: Inline AfterDelayFromFirstElementStateMachine to 
AfterProcessingTimeStateMachine and simplify
 Key: BEAM-1486
 URL: https://issues.apache.org/jira/browse/BEAM-1486
 Project: Beam
  Issue Type: Improvement
  Components: runner-core
Reporter: Kenneth Knowles
Priority: Minor


Currently, the runners/core-java implementation of processing time and 
synchronized processing time triggers share code that is actually unused and 
unsupported for synchronized processing time. It clarified things to just 
separate them in the SDK; we should do the same for the execution in 
runner/core-java.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2004: Add unsigned 64 bit int read/write methods to cytho...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2004

Add unsigned 64 bit int read/write methods to cythonized stream

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam py_coder_cy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2004.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2004


commit a08967b32ab7482c70017e2824a226869d6f4866
Author: Vikas Kedigehalli 
Date:   2017-02-14T06:11:25Z

Add unsigned 64 bit int read/write methods to cythonized stream




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1461) duplication with StartBundle and prepareForProcessing in DoFn

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866377#comment-15866377
 ] 

Xu Mingmin commented on BEAM-1461:
--

+1 `prepareForProcessing` is only used with Aggregator. It makes sense to 
deprecate according to BEAM-775 

> duplication with StartBundle and prepareForProcessing in DoFn
> -
>
> Key: BEAM-1461
> URL: https://issues.apache.org/jira/browse/BEAM-1461
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Davor Bonaci
>
> There're one annotation `StartBundle`, and one public function 
> `prepareForProcessing` in DoFn, which are called both before 
> `ProcessElement`. It's confused which one should be implemented in a subclass.
> The call sequence seems as:
> prepareForProcessing -> StartBundle -> processElement



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1485) pipeline_test:test_memory_usage is flaky

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1485:
-

 Summary: pipeline_test:test_memory_usage is flaky
 Key: BEAM-1485
 URL: https://issues.apache.org/jira/browse/BEAM-1485
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Ahmet Altay
Priority: Minor


The test is flaky, because it is sensitive to minor memory variations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1485) pipeline_test:test_memory_usage is flaky

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866341#comment-15866341
 ] 

ASF GitHub Bot commented on BEAM-1485:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/2003

[BEAM-1485] Add some slack to the memory threshold for DirectRunner oom 
checker

R: @chamikaramj 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam flake2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2003






> pipeline_test:test_memory_usage is flaky
> 
>
> Key: BEAM-1485
> URL: https://issues.apache.org/jira/browse/BEAM-1485
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>
> The test is flaky, because it is sensitive to minor memory variations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2003: [BEAM-1485] Add some slack to the memory threshold ...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/2003

[BEAM-1485] Add some slack to the memory threshold for DirectRunner oom 
checker

R: @chamikaramj 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam flake2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2003






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1469) Estimate Pi test is flaky

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1469.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Estimate Pi test is flaky
> -
>
> Key: BEAM-1469
> URL: https://issues.apache.org/jira/browse/BEAM-1469
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>
> I get this error sometimes:
> DataflowAssertException: Failed assert: 3.150432 not in [3.13, 3.15] 
> [while running 'assert_that/Match']
> root: WARNING: A task failed with exception.
>  Failed assert: 3.150432 not in [3.13, 3.15] [while running 
> 'assert_that/Match']
> This is probably related to the reduced trial size, We can relax the 
> upper/lower bounds a little bit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1484) Add post commit test for cythonized tests

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1484:
-

 Summary: Add post commit test for cythonized tests
 Key: BEAM-1484
 URL: https://issues.apache.org/jira/browse/BEAM-1484
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Ahmet Altay






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #2634

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2633

[Apache Jenkins Server]

See 

--
[...truncated 11052 lines...]
2017-02-14T10:52:00.758 [ERROR] * @param 
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
:51:
 warning: no description for @param
2017-02-14T10:52:00.758 [ERROR] * @param 
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
:52:
 warning: no description for @param
2017-02-14T10:52:00.758 [ERROR] * @param 
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
:409:
 error: bad use of '>'
2017-02-14T10:52:00.758 [ERROR] * Snapshots the state (stateName -> (valueCoder 
&& (namespace -> value)))
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
:409:
 error: bad HTML entity
2017-02-14T10:52:00.758 [ERROR] * Snapshots the state (stateName -> (valueCoder 
&& (namespace -> value)))
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
:409:
 error: bad HTML entity
2017-02-14T10:52:00.758 [ERROR] * Snapshots the state (stateName -> (valueCoder 
&& (namespace -> value)))
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
:409:
 error: bad use of '>'
2017-02-14T10:52:00.758 [ERROR] * Snapshots the state (stateName -> (valueCoder 
&& (namespace -> value)))
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
:435:
 error: bad use of '>'
2017-02-14T10:52:00.758 [ERROR] * Restore the state (stateName -> (valueCoder 
&& (namespace -> value)))
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
:435:
 error: bad HTML entity
2017-02-14T10:52:00.758 [ERROR] * Restore the state (stateName -> (valueCoder 
&& (namespace -> value)))
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
:435:
 error: bad HTML entity
2017-02-14T10:52:00.758 [ERROR] * Restore the state (stateName -> (valueCoder 
&& (namespace -> value)))
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
:435:
 error: bad use of '>'
2017-02-14T10:52:00.758 [ERROR] * Restore the state (stateName -> (valueCoder 
&& (namespace -> value)))
2017-02-14T10:52:00.758 [ERROR] ^
2017-02-14T10:52:00.758 [ERROR] 
2017-02-14T10:52:00.758 [ERROR] Command line was: 
/usr/local/asfpackages/java/jdk1.8.0_121/jre/../bin/javadoc @options @packages
2017-02-14T10:52:00.758 [ERROR] 
2017-02-14T10:52:00.758 [ERROR] Refer to the generated Javadoc files in 
'
 dir.
2017-02-14T10:52:00.758 [ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.maven.plugins:maven-javadoc-plugin:2.10.4:jar (javadoc) on project 
beam-runners-flink_2.10: MavenReportException: Error while generating Javadoc: 
Exit code: 1 - 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Dataflow #2289

[Apache Jenkins Server]

See