[jira] [Commented] (BEAM-289) Examples Use TypeDescriptors

2016-05-26 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302568#comment-15302568
 ] 

ASF GitHub Bot commented on BEAM-289:
-

GitHub user eljefe6a opened a pull request:

https://github.com/apache/incubator-beam/pull/393

[BEAM-289] Examples Use TypeDescriptorsNew type descriptors examples

Jira issue BEAM-289 Examples Use TypeDescriptors. Changed example code to 
use TypeDescriptors.

These could have used static imports to make the lines shorter, but we 
opted for the more understandable syntax.

cc: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eljefe6a/incubator-beam 
NewTypeDescriptorsExamples

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/393.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #393


commit 72ec9b82fb21ee9040b1bee6615fecfb9916e470
Author: Jesse Anderson 
Date:   2016-05-02T18:39:26Z

Make Regex Transform

commit e6f8c958a2bb88d8b2582cb9d4391922c15b7141
Author: Jesse Anderson 
Date:   2016-05-02T22:34:08Z

Merge remote-tracking branch 'upstream/master'

commit 587eaaec106829002df5df1b38753f811649aa51
Author: Jesse Anderson 
Date:   2016-05-03T01:08:13Z

Fixing checkstyle issues. Added missing Apache license.

commit df3045f62c939ef3a777ffbf658088f193144983
Author: Jesse Anderson 
Date:   2016-05-05T15:46:14Z

Added distributed replacement functions. Add replaceAll and replaceFirst. 
Fixed some JavaDocs.

commit 793d22667f485a5cdd49a7d36553c96e6898391c
Author: Jesse Anderson 
Date:   2016-05-05T15:55:58Z

Whitespace fixes for check style.

commit 9e5a9971131721c988242400643712f5c9671b9e
Author: Jesse Anderson 
Date:   2016-05-09T15:47:56Z

Merge remote-tracking branch 'upstream/master'

commit 425a4d89692f33f99a68aed511270de0ff9db4ac
Author: Jesse Anderson 
Date:   2016-05-11T19:51:57Z

Merge remote-tracking branch 'upstream/master'

commit 225b2d0ac2d8808da7756f915cbeb7684e4951a4
Author: Jesse Anderson 
Date:   2016-05-14T01:20:12Z

Merge remote-tracking branch 'upstream/master'

commit c1a3bc55b47e5dbd11c2bcb360a2a7f2c4aacfb9
Author: Jesse Anderson 
Date:   2016-05-16T20:58:08Z

Changed Word Counts to use TypeDescriptors.

commit adfeb01bafb1c17c6c65b30c45ea3e42473a80e6
Author: Jesse Anderson 
Date:   2016-05-16T21:09:18Z

Updated complete examples to use TypeDescriptors.

commit 2a455f3ae029e30644cc7b6c96105f761f903e74
Author: Jesse Anderson 
Date:   2016-05-16T22:11:57Z

Removing Regex transforms from this branch.

commit eac10506c7113b0493e972e98ef7b67fb1466fb4
Author: Jesse Anderson 
Date:   2016-05-17T02:38:42Z

Merge remote-tracking branch 'upstream/master' into TypeDescriptorsExamples

commit 0d5d9b6ccf54442345ed432c915469115a146e43
Author: Jesse Anderson 
Date:   2016-05-23T16:14:33Z

Merge commit 'bb086b8d367e2c360d965659b094035cf01d9959' into 
TypeDescriptorsExamples

commit 8ed14ca1986d45d49a43ac43a7fdf47a377103aa
Author: Jesse Anderson 
Date:   2016-05-23T19:08:12Z

Trivial change to kick off another build.

commit f9cd71989cf23d12ba0462c38d2e212a54902a4b
Author: Jesse Anderson 
Date:   2016-05-23T19:08:34Z

Trivial change to kick off another build.

commit fae3af03e3ae30ecc90f015833daab4d2e8868ef
Author: Jesse Anderson 
Date:   2016-05-02T18:39:26Z

Make Regex Transform

commit f098c5e550bfdd50e7694727fd8f4e7156a9070c
Author: Jesse Anderson 
Date:   2016-05-03T01:08:13Z

Fixing checkstyle issues. Added missing Apache license.

commit 78f4f531c5898188ae9435fe3cd39fe589395811
Author: Jesse Anderson 
Date:   2016-05-05T15:46:14Z

Added distributed replacement functions. Add replaceAll and replaceFirst. 
Fixed some JavaDocs.

commit d812294124192e4acbcfc526472ef8dff6e98433
Author: Jesse Anderson 
Date:   2016-05-05T15:55:58Z

Whitespace fixes for check style.

commit 8c43cb27f6047edab9e450f9b84cfaf478aeb470
Author: Jesse Anderson 
Date:   2016-05-16T20:58:08Z

Changed Word Counts to use TypeDescriptors.

commit 6834cbefa08f4ea07e932bebba43bd17421a5d1a
Author: Jesse Anderson 
Date:   2016-05-16T21:09:18Z

Updated complete examples to use TypeDescriptors.

commit b5ad893fd3a3768bbd341f399eae87509a1c0391
Author: Jesse Anderson 
Date:   2016-05-16T22:11:57Z

Removing Regex transforms from this branch.

commit 3c942925f485ecac014e6c1b1b669e16813b5f62
Author: Jesse Anderson 
Date:   2016-05-23T19:08:12Z

Trivial change to kick off another build.

commit d7b86d82c320163560a9a2ba5436ded88f770fed
Author: Jesse Anderson 
Date:   2016-05-23T19:08:34Z

Trivial change to kick off another build.

commit 089fba1f86823b368c0df7b8844cc1f40014bc63
Author: Jesse Anderson 
Date:   2016-05-24T04:50:49Z

M

[GitHub] incubator-beam pull request: [BEAM-289] Examples Use TypeDescripto...

2016-05-26 Thread eljefe6a
GitHub user eljefe6a opened a pull request:

https://github.com/apache/incubator-beam/pull/393

[BEAM-289] Examples Use TypeDescriptorsNew type descriptors examples

Jira issue BEAM-289 Examples Use TypeDescriptors. Changed example code to 
use TypeDescriptors.

These could have used static imports to make the lines shorter, but we 
opted for the more understandable syntax.

cc: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eljefe6a/incubator-beam 
NewTypeDescriptorsExamples

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/393.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #393


commit 72ec9b82fb21ee9040b1bee6615fecfb9916e470
Author: Jesse Anderson 
Date:   2016-05-02T18:39:26Z

Make Regex Transform

commit e6f8c958a2bb88d8b2582cb9d4391922c15b7141
Author: Jesse Anderson 
Date:   2016-05-02T22:34:08Z

Merge remote-tracking branch 'upstream/master'

commit 587eaaec106829002df5df1b38753f811649aa51
Author: Jesse Anderson 
Date:   2016-05-03T01:08:13Z

Fixing checkstyle issues. Added missing Apache license.

commit df3045f62c939ef3a777ffbf658088f193144983
Author: Jesse Anderson 
Date:   2016-05-05T15:46:14Z

Added distributed replacement functions. Add replaceAll and replaceFirst. 
Fixed some JavaDocs.

commit 793d22667f485a5cdd49a7d36553c96e6898391c
Author: Jesse Anderson 
Date:   2016-05-05T15:55:58Z

Whitespace fixes for check style.

commit 9e5a9971131721c988242400643712f5c9671b9e
Author: Jesse Anderson 
Date:   2016-05-09T15:47:56Z

Merge remote-tracking branch 'upstream/master'

commit 425a4d89692f33f99a68aed511270de0ff9db4ac
Author: Jesse Anderson 
Date:   2016-05-11T19:51:57Z

Merge remote-tracking branch 'upstream/master'

commit 225b2d0ac2d8808da7756f915cbeb7684e4951a4
Author: Jesse Anderson 
Date:   2016-05-14T01:20:12Z

Merge remote-tracking branch 'upstream/master'

commit c1a3bc55b47e5dbd11c2bcb360a2a7f2c4aacfb9
Author: Jesse Anderson 
Date:   2016-05-16T20:58:08Z

Changed Word Counts to use TypeDescriptors.

commit adfeb01bafb1c17c6c65b30c45ea3e42473a80e6
Author: Jesse Anderson 
Date:   2016-05-16T21:09:18Z

Updated complete examples to use TypeDescriptors.

commit 2a455f3ae029e30644cc7b6c96105f761f903e74
Author: Jesse Anderson 
Date:   2016-05-16T22:11:57Z

Removing Regex transforms from this branch.

commit eac10506c7113b0493e972e98ef7b67fb1466fb4
Author: Jesse Anderson 
Date:   2016-05-17T02:38:42Z

Merge remote-tracking branch 'upstream/master' into TypeDescriptorsExamples

commit 0d5d9b6ccf54442345ed432c915469115a146e43
Author: Jesse Anderson 
Date:   2016-05-23T16:14:33Z

Merge commit 'bb086b8d367e2c360d965659b094035cf01d9959' into 
TypeDescriptorsExamples

commit 8ed14ca1986d45d49a43ac43a7fdf47a377103aa
Author: Jesse Anderson 
Date:   2016-05-23T19:08:12Z

Trivial change to kick off another build.

commit f9cd71989cf23d12ba0462c38d2e212a54902a4b
Author: Jesse Anderson 
Date:   2016-05-23T19:08:34Z

Trivial change to kick off another build.

commit fae3af03e3ae30ecc90f015833daab4d2e8868ef
Author: Jesse Anderson 
Date:   2016-05-02T18:39:26Z

Make Regex Transform

commit f098c5e550bfdd50e7694727fd8f4e7156a9070c
Author: Jesse Anderson 
Date:   2016-05-03T01:08:13Z

Fixing checkstyle issues. Added missing Apache license.

commit 78f4f531c5898188ae9435fe3cd39fe589395811
Author: Jesse Anderson 
Date:   2016-05-05T15:46:14Z

Added distributed replacement functions. Add replaceAll and replaceFirst. 
Fixed some JavaDocs.

commit d812294124192e4acbcfc526472ef8dff6e98433
Author: Jesse Anderson 
Date:   2016-05-05T15:55:58Z

Whitespace fixes for check style.

commit 8c43cb27f6047edab9e450f9b84cfaf478aeb470
Author: Jesse Anderson 
Date:   2016-05-16T20:58:08Z

Changed Word Counts to use TypeDescriptors.

commit 6834cbefa08f4ea07e932bebba43bd17421a5d1a
Author: Jesse Anderson 
Date:   2016-05-16T21:09:18Z

Updated complete examples to use TypeDescriptors.

commit b5ad893fd3a3768bbd341f399eae87509a1c0391
Author: Jesse Anderson 
Date:   2016-05-16T22:11:57Z

Removing Regex transforms from this branch.

commit 3c942925f485ecac014e6c1b1b669e16813b5f62
Author: Jesse Anderson 
Date:   2016-05-23T19:08:12Z

Trivial change to kick off another build.

commit d7b86d82c320163560a9a2ba5436ded88f770fed
Author: Jesse Anderson 
Date:   2016-05-23T19:08:34Z

Trivial change to kick off another build.

commit 089fba1f86823b368c0df7b8844cc1f40014bc63
Author: Jesse Anderson 
Date:   2016-05-24T04:50:49Z

Merge branch 'TypeDescriptorsExamples' of 
https://github.com/eljefe6a/incubator-beam into TypeDescriptorsExamples




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project 

[jira] [Commented] (BEAM-289) Examples Use TypeDescriptors

2016-05-26 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302562#comment-15302562
 ] 

ASF GitHub Bot commented on BEAM-289:
-

Github user eljefe6a closed the pull request at:

https://github.com/apache/incubator-beam/pull/342


> Examples Use TypeDescriptors
> 
>
> Key: BEAM-289
> URL: https://issues.apache.org/jira/browse/BEAM-289
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Jesse Anderson
>Assignee: Frances Perry
>
> Change the Java and Java 8 examples to use TypeDescriptors instead of inline 
> TypeDescriptor creation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] incubator-beam pull request: [BEAM-289] Examples Use TypeDescripto...

2016-05-26 Thread eljefe6a
Github user eljefe6a closed the pull request at:

https://github.com/apache/incubator-beam/pull/342


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Updated] (BEAM-170) Session windows should not be identified by their bounds

2016-05-26 Thread Kenneth Knowles (JIRA)

 [ 
https://issues.apache.org/jira/browse/BEAM-170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-170:
-
Assignee: Ilya Ganelin  (was: Davor Bonaci)

> Session windows should not be identified by their bounds
> 
>
> Key: BEAM-170
> URL: https://issues.apache.org/jira/browse/BEAM-170
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Ilya Ganelin
>
> Today, if two session windows for the same key have the same bounds, they are 
> considered the same window. This is an accident. It is not intended that any 
> session windows are considered equal except via the operation of merging them 
> into the same session.
> A risk associated with this behavior is that two windows that happen to 
> coincide will share per-window-and-key state rather than evolving separately 
> and having their separate state reconciled by state merging logic. These code 
> paths are not required to be coherent, and in practice they are not.
> In particular, if the trigger for a session window ever finishes, then 
> subsequent data in a window with the same bounds will be dropped, whereas if 
> it had differed by a millisecond it would have created a new session, 
> ignoring the previously closed session.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (BEAM-170) Session windows should not be identified by their bounds

2016-05-26 Thread Kenneth Knowles (JIRA)

 [ 
https://issues.apache.org/jira/browse/BEAM-170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-170:
-
Assignee: Davor Bonaci

> Session windows should not be identified by their bounds
> 
>
> Key: BEAM-170
> URL: https://issues.apache.org/jira/browse/BEAM-170
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Davor Bonaci
>
> Today, if two session windows for the same key have the same bounds, they are 
> considered the same window. This is an accident. It is not intended that any 
> session windows are considered equal except via the operation of merging them 
> into the same session.
> A risk associated with this behavior is that two windows that happen to 
> coincide will share per-window-and-key state rather than evolving separately 
> and having their separate state reconciled by state merging logic. These code 
> paths are not required to be coherent, and in practice they are not.
> In particular, if the trigger for a session window ever finishes, then 
> subsequent data in a window with the same bounds will be dropped, whereas if 
> it had differed by a millisecond it would have created a new session, 
> ignoring the previously closed session.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (BEAM-170) Session windows should not be identified by their bounds

2016-05-26 Thread Ilya Ganelin (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302423#comment-15302423
 ] 

Ilya Ganelin commented on BEAM-170:
---

I'd be happy to work on this. 

> Session windows should not be identified by their bounds
> 
>
> Key: BEAM-170
> URL: https://issues.apache.org/jira/browse/BEAM-170
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> Today, if two session windows for the same key have the same bounds, they are 
> considered the same window. This is an accident. It is not intended that any 
> session windows are considered equal except via the operation of merging them 
> into the same session.
> A risk associated with this behavior is that two windows that happen to 
> coincide will share per-window-and-key state rather than evolving separately 
> and having their separate state reconciled by state merging logic. These code 
> paths are not required to be coherent, and in practice they are not.
> In particular, if the trigger for a session window ever finishes, then 
> subsequent data in a window with the same bounds will be dropped, whereas if 
> it had differed by a millisecond it would have created a new session, 
> ignoring the previously closed session.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (BEAM-170) Session windows should not be identified by their bounds

2016-05-26 Thread Kenneth Knowles (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302336#comment-15302336
 ] 

Kenneth Knowles commented on BEAM-170:
--

Yes, creating a new type of window for sessions that includes a UUID is a 
natural solution. Would you like to take this ticket? :-)

> Session windows should not be identified by their bounds
> 
>
> Key: BEAM-170
> URL: https://issues.apache.org/jira/browse/BEAM-170
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>
> Today, if two session windows for the same key have the same bounds, they are 
> considered the same window. This is an accident. It is not intended that any 
> session windows are considered equal except via the operation of merging them 
> into the same session.
> A risk associated with this behavior is that two windows that happen to 
> coincide will share per-window-and-key state rather than evolving separately 
> and having their separate state reconciled by state merging logic. These code 
> paths are not required to be coherent, and in practice they are not.
> In particular, if the trigger for a session window ever finishes, then 
> subsequent data in a window with the same bounds will be dropped, whereas if 
> it had differed by a millisecond it would have created a new session, 
> ignoring the previously closed session.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (BEAM-122) GlobalWindow and allowedLateness can cause inconsistent timer interpretation

2016-05-26 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302315#comment-15302315
 ] 

ASF GitHub Bot commented on BEAM-122:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/391


> GlobalWindow and allowedLateness can cause inconsistent timer interpretation 
> -
>
> Key: BEAM-122
> URL: https://issues.apache.org/jira/browse/BEAM-122
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Mark Shields
>Assignee: Mark Shields
>
> In ReduceFnRunner we have code such as
>window.getMaxTimestamp().plus(windowingStrategy.getAllowedLateness())
> If window is global then maxTimestamp will be 
> BoundedWindow.TIMESTAMP_MAX_VALUE.
> Meanwhile, timestamps beyond BoundedWindow.TIMESTAMP_MAX_VALUE will be 
> clipped in most runners.
> This could cause the time of an expected timer (eg for garbage collection) to 
> not match the actual time of a fired timer.
> We should either make non-zero allowedLateness on the Global window illegal 
> (probably obnoxious) or explicitly clip it to zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] incubator-beam pull request: [BEAM-122][BEAM-311] Add GC hold if h...

2016-05-26 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/391


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[1/2] incubator-beam git commit: Add GC hold if have data. Don't set timers beyond GlobalWindow.maxTimestamp

2016-05-26 Thread kenn
Repository: incubator-beam
Updated Branches:
  refs/heads/master d4c052c32 -> cca2577c6


Add GC hold if have data. Don't set timers beyond GlobalWindow.maxTimestamp


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9ab5b888
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9ab5b888
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9ab5b888

Branch: refs/heads/master
Commit: 9ab5b888998ace0fb9b8396dc0355edd44bc65f8
Parents: d4c052c
Author: Mark Shields 
Authored: Tue May 24 13:52:41 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu May 26 09:06:12 2016 -0700

--
 .../apache/beam/sdk/util/ReduceFnRunner.java| 106 +++---
 .../org/apache/beam/sdk/util/WatermarkHold.java | 212 ---
 .../beam/sdk/util/ReduceFnRunnerTest.java   | 113 ++
 3 files changed, 325 insertions(+), 106 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9ab5b888/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReduceFnRunner.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReduceFnRunner.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReduceFnRunner.java
index e916aa8..889ac6f 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReduceFnRunner.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReduceFnRunner.java
@@ -22,6 +22,7 @@ import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.transforms.windowing.OutputTimeFn;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
@@ -580,14 +581,17 @@ public class ReduceFnRunner {
   "ReduceFnRunner.onTimer: Note that timer {} is for non-ACTIVE window 
{}", timer, window);
 }
 
-// If this is an end-of-window timer then, we need to set a GC timer
+// If this is an end-of-window timer then we may need to set a garbage 
collection timer
+// if allowed lateness is non-zero.
 boolean isEndOfWindow = TimeDomain.EVENT_TIME == timer.getDomain()
-  && timer.getTimestamp().equals(window.maxTimestamp());
+&& timer.getTimestamp().equals(window.maxTimestamp());
 
 // If this is a garbage collection timer then we should trigger and 
garbage collect the window.
-Instant cleanupTime = 
window.maxTimestamp().plus(windowingStrategy.getAllowedLateness());
-boolean isGarbageCollection =
-TimeDomain.EVENT_TIME == timer.getDomain() && 
timer.getTimestamp().equals(cleanupTime);
+// We'll consider any timer at or after the end-of-window time to be a 
signal to garbage
+// collect.
+Instant cleanupTime = garbageCollectionTime(window);
+boolean isGarbageCollection = TimeDomain.EVENT_TIME == timer.getDomain()
+&& !timer.getTimestamp().isBefore(cleanupTime);
 
 if (isGarbageCollection) {
   WindowTracing.debug(
@@ -600,7 +604,9 @@ public class ReduceFnRunner {
 // We need to call onTrigger to emit the final pane if required.
 // The final pane *may* be ON_TIME if no prior ON_TIME pane has been 
emitted,
 // and the watermark has passed the end of the window.
-onTrigger(directContext, renamedContext, true/* isFinished */, 
isEndOfWindow);
+@Nullable Instant newHold =
+onTrigger(directContext, renamedContext, true/* isFinished */, 
isEndOfWindow);
+Preconditions.checkState(newHold == null);
   }
 
   // Cleanup flavor B: Clear all the remaining state for this window since 
we'll never
@@ -626,7 +632,7 @@ public class ReduceFnRunner {
 // timer. (If getAllowedLateness is zero then the timer event will be 
considered a
 // cleanup event and handled by the above).
 // Note we must do this even if the trigger is finished so that we are 
sure to cleanup
-// any final trigger tombstones.
+// any final trigger finished bits.
 Preconditions.checkState(
 windowingStrategy.getAllowedLateness().isLongerThan(Duration.ZERO),
 "Unexpected zero getAllowedLateness");
@@ -635,6 +641,8 @@ public class ReduceFnRunner {
 + "inputWatermark:{}; outputWatermark:{}",
 key, directContext.window(), cleanupTime, 
timerInternals.currentInputWatermarkTime(),
 timerInternals.currentOutputWatermarkTime());
+
Preconditions.checkState(!cleanupTime.isAfter(BoundedWindow.TIMESTAMP_MAX_VALUE),
+ 

[2/2] incubator-beam git commit: This closes #391

2016-05-26 Thread kenn
This closes #391


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cca2577c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cca2577c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cca2577c

Branch: refs/heads/master
Commit: cca2577c626714b4f351adbd5c1e8f5dbb5b852e
Parents: d4c052c 9ab5b88
Author: Kenneth Knowles 
Authored: Thu May 26 09:06:45 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu May 26 09:06:45 2016 -0700

--
 .../apache/beam/sdk/util/ReduceFnRunner.java| 106 +++---
 .../org/apache/beam/sdk/util/WatermarkHold.java | 212 ---
 .../beam/sdk/util/ReduceFnRunnerTest.java   | 113 ++
 3 files changed, 325 insertions(+), 106 deletions(-)
--