[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767897#comment-15767897 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1660 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767732#comment-15767732 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1673 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767452#comment-15767452 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1668 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15766056#comment-15766056 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1673 [BEAM-27] Require TimeDomain to delete a timer Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @aljoscha A bit of an oversight, I neglected the fact that runners generally store different sorts of timers in rather different ways. When a user sets a timer, the `DoFnSignature` is available, so this will be for free. And when system code deletes a timer, the domain will always be known. This will require a Dataflow update, so don't worry if Dataflow-specific integration tests don't pass. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam delete-by-domain Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1673.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1673 commit 46dfd0fb4d2a1533d3ed053983faee6537d3ccf0 Author: Kenneth KnowlesDate: 2016-12-21T04:09:25Z Require TimeDomain to delete a timer > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765387#comment-15765387 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1668 [BEAM-27, BEAM-362] Remove deprecated InMemoryTimerInternals from SDK Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam InMemoryTimerInternals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1668.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1668 commit bbea8469912b23383a9ae5cf084b5801706e Author: Kenneth KnowlesDate: 2016-12-20T22:07:00Z Remove deprecated InMemoryTimerInternals from SDK > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765055#comment-15765055 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1652 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762533#comment-15762533 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1660 [BEAM-27] Support setting and deleting timers by ID in InMemoryTimerInternals Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This is build on top of the move to runners-core in #1652. Only the second commit contains nontrivial changes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam InMemoryTimerInternals-dedup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1660.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1660 commit 6effd6c93587d1c0a02ba9285b47d165ae5c517d Author: Kenneth KnowlesDate: 2016-12-16T04:45:56Z Move InMemoryTimerInternals to runners-core commit 557d2d724c53233ad9d34c9239ff5cf77b754d73 Author: Kenneth Knowles Date: 2016-12-17T04:22:59Z Restore SDK's InMemoryTimerInternals, deprecated commit 0ef0e3a3ecabafdd934f7d03f47f0a220187fb22 Author: Kenneth Knowles Date: 2016-12-19T22:01:36Z Support set and delete of timer by ID in InMemoryTimerInternals > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15757625#comment-15757625 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1612 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747289#comment-15747289 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1612 [BEAM-27] Support timer setting and receiving in SimpleDoFnRunner Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- These are a couple commits peeled off that should enable each runner to begin fulfilling the timer API. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam SimpleDoFnRunner-timers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1612.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1612 commit 95f954de00d20b27e127e41a0118a305e2cd8a94 Author: Kenneth KnowlesDate: 2016-12-08T04:09:06Z Make TimerSpec and StateSpec fields accessible commit 19ccebe6170b24af0067c7a04483d8e9156a0ef5 Author: Kenneth Knowles Date: 2016-11-23T22:21:40Z Add timer support to DoFnRunner(s) > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733347#comment-15733347 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1555 [BEAM-27] Fix exclusion of UsesTimersInParDo for FlinkRunner streaming tests Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: anyone (fixing postcommit) ping @aljoscha @mxm You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam FlinkRunner Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1555.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1555 commit a11eb637f6f1c7bdc41b24154cb3893bb194b096 Author: Kenneth KnowlesDate: 2016-12-08T20:42:03Z Fix exclusion of UsesTimersInParDo for FlinkRunner streaming tests > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15732940#comment-15732940 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1550 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15731074#comment-15731074 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1550 [BEAM-27] Reject timers for ParDo in each runner separately and exclude timer tests Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- There are many trivial commits spread across the runners, in two phases. 1. Add a JUnit category for timers and exclude it from all runners. I've included one trivial test just to see that it is working. (FWIW I have hacked the direct runner enough in later commits to get this test to pass "for the wrong reasons") 2. Remove the timer rejection code from `ParDo` and add it to every runner. R: @tgroh @aljoscha @amitsela @tweise I'll wait for all four to LGTM. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam UsesTimers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1550.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1550 commit 5c99d2c3a573c3963fd6efa9e11ed860d26f0365 Author: Kenneth KnowlesDate: 2016-12-07T04:49:15Z Add JUnit category UsesTimersInParDo With this, various runners can disable tests for this capability until they support it. commit 604695f1b7cbd6dfeda94d82532fde535b3e3448 Author: Kenneth Knowles Date: 2016-12-07T04:52:24Z Disable tests for timers in ParDo for Apex runner commit 58cac2f4960044e646249eb8a3db27b345e1 Author: Kenneth Knowles Date: 2016-12-07T04:52:49Z Disable tests for timers in ParDo for Flink runner commit f27bdf9b3c2e609c54615db60f0682b34f6575c5 Author: Kenneth Knowles Date: 2016-12-07T04:53:05Z Disables tests for timers in ParDo for Spark runner commit be9c04bf0f2af7b428a61d3b74b68ea18c4f0e03 Author: Kenneth Knowles Date: 2016-12-07T04:53:16Z Disable tests for timers in ParDo for Dataflow runner commit b66c5cb4f92e2c0d834f44ba246b8186cc9d0291 Author: Kenneth Knowles Date: 2016-12-08T04:24:34Z Disable tests for timers in ParDo for direct runner commit e4f062495878328359234da4e586d12cb42965f9 Author: Kenneth Knowles Date: 2016-12-07T04:49:40Z Add basic test for timers in ParDoTest commit ad4cbbc18ffbbedaaa17415220d7ff2908931797 Author: Kenneth Knowles Date: 2016-12-08T04:04:51Z No longer reject timers in ParDo commit 0703093767db2776965384b09b322f01f4b79b39 Author: Kenneth Knowles Date: 2016-12-08T04:34:34Z Reject timers for ParDo in ApexRunner commit 4b312e8d01e324db77e5f2de62eae35efc153adb Author: Kenneth Knowles Date: 2016-12-08T04:34:59Z Reject timers for ParDo in FlinkRunner commit 2e2bc885deb5000a269800bdbead8f17dff5a3a9 Author: Kenneth Knowles Date: 2016-12-08T04:35:08Z Reject timers for ParDo in SparkRunner commit 2d2d83f721c21f681f431974edc4e044d44828c5 Author: Kenneth Knowles Date: 2016-12-08T04:37:33Z Reject timers for ParDo in DirectRunner > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730970#comment-15730970 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1528 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727629#comment-15727629 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1528 [BEAM-27] Add DoFn.OnTimerContext and support as a DoFn parameter Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam OnTimerContext Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1528.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1528 commit 44a7b915d502c318ffabaa6fb808207bb3ea15e8 Author: Kenneth KnowlesDate: 2016-12-07T04:10:06Z Add DoFn.OnTimerContext commit 1934b704411fed76a58cbc657c7dc8be666c3885 Author: Kenneth Knowles Date: 2016-12-07T04:01:18Z Add support for OnTimerContext parameter in DoFnSignature commit 14d9a3794c4f8c68626075343a1dec1d4c017686 Author: Kenneth Knowles Date: 2016-12-07T04:10:21Z Access to OnTimerContext via DoFnInvokers.ArgumentProvider > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716579#comment-15716579 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1477 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710850#comment-15710850 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1477 [BEAM-27] Add timerId to TimerData Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This `timerId` is generated to be identical to historical behavior, and to be unique per time domain and timestamp. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam timerId Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1477.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1477 commit d53275d3075d1491d8718837b00093be09f93354 Author: Kenneth KnowlesDate: 2016-11-23T22:30:57Z Add timerId to TimerData This timerId is generated to be identical to historical behavior, and to be unique per time domain and timestamp. > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: New Feature > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669463#comment-15669463 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1307 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646348#comment-15646348 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1307 [BEAM-27] Connect generated DoFnInvoker's invokerOnTimer to generated OnTimerInvoker Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DoFnInvokers-OnTimerInvokers Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1307.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1307 commit 3e759d3be013fe38cea2601fe4f7cfe95b41d42b Author: Kenneth KnowlesDate: 2016-11-02T04:23:48Z Connect generated DoFnInvoker.invokerOnTimer to OnTimerInvoker > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646243#comment-15646243 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1256 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637031#comment-15637031 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1252 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15626917#comment-15626917 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1252 [BEAM-27] Factor of parameter analysis in DoFnSignatures; use it for @OnTimer methods Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @jkff This might be easiest to review as individual commits, though the last one is the goal of this PR. This PR is a prefix of ongoing work to invoke `@OnTimer` methods. I've peeled it off and fixed it up and added a smidge of testing. Since the code is entirely shared, just making sure the right path is taken provides moderate coverage. (a further refactor could make this clearer, by isolating the shared code into a class with its own unit tests, absolving its callers of proving they exercise the code path, but I thought this baby step was better) You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam DoFnSignatures Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1252.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1252 commit 5946ec175d55f9164f9b1b34a06539da3c24f5ae Author: Kenneth KnowlesDate: 2016-11-01T04:30:40Z Refactor and reuse parameter analysis in DoFnSignatures commit 15a8901ade07c1649378a0b72238eb514877c61c Author: Kenneth Knowles Date: 2016-11-01T21:48:54Z Switch DoFnSignature, etc, from TypeToken to TypeDescriptor commit 6de216695c05e4d9b283f0842aed8713075be7aa Author: Kenneth Knowles Date: 2016-11-01T21:50:24Z DoFnSignature: Make TypeDescriptor-returning methods public commit 40bbfff3dba803951607ae15d5d2cf99fbe64a32 Author: Kenneth Knowles Date: 2016-11-01T22:38:01Z Add TypeDescriptor#getTypes > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621078#comment-15621078 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1217 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615438#comment-15615438 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1217 [BEAM-27] Add setTimer and deleteTimer by ID in TimerInternals Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- I have pulled this commit out of my ongoing work to add timers to `DoFn`, specifically my branch where I add `DoFnInvoker.invokeOnTimer(, ...)`. For users, timers will have IDs. This is a step towards that API. I think it is also just better in most ways, especially for being able to delete timers without having to know exactly when it was set for. Before this change, a timer's ID is its `TimeDomain` + `Instant`. Inlining `TimerData` for this description, I think `setTimer(TimeDomain, Instant)` is OK for setting an anonymous "wake up" as we use it today, while `deleteTimer(TimeDomain, Instant)` should be removed. R: @tgroh and @aljoscha and @amitsela I am curious about runner-specific thoughts if you have any. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam TimerInternals Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1217.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1217 commit 932cb20e64199942369376748faabae548f72df4 Author: Kenneth KnowlesDate: 2016-10-25T19:28:14Z Add setTimer and deleteTimer by ID in TimerInternals For users, timers will have IDs. This is a step towards that API; our current API treats the timer's timestamp as its ID, more-or-less. We can leave that API or not. This change adds the more general API. > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592917#comment-15592917 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1136 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590403#comment-15590403 ] ASF GitHub Bot commented on BEAM-27: Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1102 > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590118#comment-15590118 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1136 [BEAM-27] Add analysis of Timer parameters for DoFn.ProcessElement Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This PR adds `Timer` parameters. It includes #1102 and #1130. The final commit is the only one that is new, and can be reviewed independently. R: @jkff I will pull in a committer later. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam Timer-params Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1136.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1136 commit 507cba8f0df44b1c11c7621f00ff492243bdd1f1 Author: Kenneth KnowlesDate: 2016-10-14T06:18:40Z Tighten up tests for StateId validation commit 08fdb304a777d981fa3860fcd0270aa40f99da75 Author: Kenneth Knowles Date: 2016-10-14T21:19:59Z Fix condition on state/splittable ParDo validation commit 65eaedbfb628f6eb42d17c16220e973b064608d7 Author: Kenneth Knowles Date: 2016-10-13T20:41:38Z Add Timer, TimerId, TimerSpec and TimerSpecs These are the simple specifications that describe a timer without instantiating it. Consists only of the time domain. commit 0898999f34270a11beddb68e5068c5258ec9ebec Author: Kenneth Knowles Date: 2016-10-14T02:53:31Z Add DoFnSignatures analysis of timer declarations commit e44e56da62e48f1d763e2b2d46d55a04fbe3596c Author: Kenneth Knowles Date: 2016-10-18T20:09:57Z Reject all timers in ParDo, for now commit bcb07b534475637d06a14695f38b1e151912 Author: Kenneth Knowles Date: 2016-10-18T21:15:34Z fixup! More caveats on Timer javadoc commit 97aaf9730915d426fd865eeb67793988a8dbdefb Author: Kenneth Knowles Date: 2016-10-18T21:16:09Z fixup! state javadoc commit ac419d245a7bd3dc9263e61567676e3d92133f9a Author: Kenneth Knowles Date: 2016-10-18T21:21:41Z fixup! refine Timer javadoc commit 28fbe1890b54e26f7c0d0bc547440af061d1e907 Author: Kenneth Knowles Date: 2016-10-18T22:52:28Z fixup! validate is now void commit dcce856a1fa3d8bde6b47cf200bd212de9d2 Author: Kenneth Knowles Date: 2016-10-14T17:51:57Z Use the correct State class in DoFnSignatures commit 70ec03e5cacd272b194589ab3d7ce89e39be7b13 Author: Kenneth Knowles Date: 2016-10-14T18:07:55Z Add analysis and validation of State parameters to DoFn.ProcessElement commit 5883e0c087f1b5642f9349ab099f2eaa82caf8fa Author: Kenneth Knowles Date: 2016-10-19T22:00:30Z Merge remote-tracking branch 'github/pr/1102' into Timer-params commit bda251e05ffe48873ccaf06f005c3847acd6085b Author: Kenneth Knowles Date: 2016-10-19T22:02:36Z Add analysis of Timer parameters to DoFn.ProcessElement > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586870#comment-15586870 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1130 [BEAM-27] Add parsing and validation of @StateId params to @ProcessElement Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @bjchambers @jkff This is an initial draft to discuss some design decisions along the way. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam State-params Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1130.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1130 commit 33c884118ed04c0a008ef6018a06fa336a866de8 Author: Kenneth KnowlesDate: 2016-10-14T17:51:57Z Use the correct State class in DoFnSignatures commit 81b6994bf8162bfc3aaba40a708ce8385be76d80 Author: Kenneth Knowles Date: 2016-10-14T18:07:55Z Add analysis and validation of State parameters to DoFn.ProcessElement > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers
[ https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574420#comment-15574420 ] ASF GitHub Bot commented on BEAM-27: GitHub user kennknowles opened a pull request: https://github.com/apache/incubator-beam/pull/1102 [BEAM-27] Add initial bits for DoFn Timer API Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- R: @jkff CC: @bjchambers who gave lots of good feedback on the `StateSpec` version of this PR. Time permitting, see if you have opinions about this simpler one. This adds the interfaces and annotations from the design for the user-facing timer API, with basic reflection and validation and some tests. (while putting this together, I noticed that there were state examples that were invalid in ways other than the way they were testing, so that commit is in this PR too) You can merge this pull request into a Git repository by running: $ git pull https://github.com/kennknowles/incubator-beam TimerSpec Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/1102.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1102 commit 238b7eaa9425247ce7152b268d488581032e5411 Author: Kenneth KnowlesDate: 2016-10-14T06:18:40Z Tighten up tests for StateId validation commit b1aa85d764c70ce648e4d5c068ce1b7f1e36a2ac Author: Kenneth Knowles Date: 2016-10-13T20:41:38Z Add Timer, TimerId, TimerSpec and TimerSpecs These are the simple specifications that describe a timer without instantiating it. Consists only of the time domain. commit 0cd37b50ee0d92886060fe9cb8e28273e24e0c13 Author: Kenneth Knowles Date: 2016-10-14T02:53:31Z Add DoFnSignatures analysis of timer declarations commit fe2cf1688068ac0731b67724c1a8f591f03ab159 Author: Kenneth Knowles Date: 2016-10-14T06:17:36Z Reject all timers in ParDo, for now > Add user-ready API for interacting with timers > -- > > Key: BEAM-27 > URL: https://issues.apache.org/jira/browse/BEAM-27 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Pipeline authors will benefit from a different factorization of interaction > with underlying timers. The current APIs are targeted at runner implementers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)