[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=229684&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-229684 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 18/Apr/19 13:44 Start Date: 18/Apr/19 13:44 Worklog Time Spent: 10m Work Description: tweise commented on pull request #8253: BEAM-7035: Remove unused parameter of clean method URL: https://github.com/apache/beam/pull/8253 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 229684) Time Spent: 4h 10m (was: 4h) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=229073&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-229073 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 17/Apr/19 13:23 Start Date: 17/Apr/19 13:23 Worklog Time Spent: 10m Work Description: tweise commented on pull request #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 229073) Time Spent: 4h (was: 3h 50m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=228775&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228775 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 17/Apr/19 01:35 Start Date: 17/Apr/19 01:35 Worklog Time Spent: 10m Work Description: tweise commented on issue #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#issuecomment-483903608 > Thanks Thomas for fixing this. This LGTM. IIUC, cancelling timers will be done by setting their timestamp beyond the maximum timestamp. Is that correct? That's right. There is currently no way to encode "cancel timer", so this is using the timestamp trick. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228775) Time Spent: 3h 50m (was: 3h 40m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=228613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228613 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 16/Apr/19 19:01 Start Date: 16/Apr/19 19:01 Worklog Time Spent: 10m Work Description: pabloem commented on issue #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#issuecomment-483802197 Thanks Thomas for fixing this. This LGTM. IIUC, cancelling timers will be done by setting their timestamp beyond the maximum timestamp. Is that correct? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228613) Time Spent: 3.5h (was: 3h 20m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=228614&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228614 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 16/Apr/19 19:01 Start Date: 16/Apr/19 19:01 Worklog Time Spent: 10m Work Description: pabloem commented on issue #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#issuecomment-483802197 Thanks Thomas for fixing this. This LGTM. IIUC, cancelling timers will be done by setting their timestamp beyond the maximum timestamp. Is that correct? FWIW, I am not very familiar with portability APIs, so I defer any portability comments to Luke and Robert, who are mucuh more familiar. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228614) Time Spent: 3h 40m (was: 3.5h) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227924&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227924 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 15/Apr/19 19:59 Start Date: 15/Apr/19 19:59 Worklog Time Spent: 10m Work Description: tweise commented on issue #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#issuecomment-483396266 @lukecwik makes sense. I will see to make the change to the test and also bring the overall topic to the ML. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227924) Time Spent: 3h 20m (was: 3h 10m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227845&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227845 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 15/Apr/19 17:19 Start Date: 15/Apr/19 17:19 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#issuecomment-483341549 Use standard_coders.yaml to perform the timer test. You can do that by copying over the missing timer definition into the python one or attempt to solve BEAM-7015 as well and migrate to use the standard_coders.yaml found underneath the model/ folder. It would be nice to discuss the discrepancy on the ML but that shouldn't block this change from going in. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227845) Time Spent: 3h 10m (was: 3h) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227806&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227806 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 15/Apr/19 16:20 Start Date: 15/Apr/19 16:20 Worklog Time Spent: 10m Work Description: tweise commented on issue #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#issuecomment-483321081 @lukecwik can you please clarify what changes you request? I assume you are referring to work that should be done to clean up this timestamp and timer mess in general. That's certainly reasonable outside of this PR, which is fixing a bug that we are blocked on. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227806) Time Spent: 3h (was: 2h 50m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227736&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227736 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 15/Apr/19 14:25 Start Date: 15/Apr/19 14:25 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#discussion_r275386072 ## File path: sdks/python/apache_beam/runners/worker/bundle_processor.py ## @@ -381,8 +381,12 @@ def set(self, ts): windowed_value.WindowedValue( (self._key, dict(timestamp=ts)), ts, (self._window,))) - def clear(self, timestamp): -self._receiver.receive((self._key, dict(clear=True))) + def clear(self): +dummy_millis = int(common_urns.constants.MAX_TIMESTAMP_MILLIS.constant) + 1 Review comment: Ok, max timestamp is then `Long.MAX_VALUE / 1000`. Seems safe to add 1 to it then. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227736) Time Spent: 2h 50m (was: 2h 40m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[
https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227662&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227662
]
ASF GitHub Bot logged work on BEAM-7035:
Author: ASF GitHub Bot
Created on: 15/Apr/19 13:06
Start Date: 15/Apr/19 13:06
Worklog Time Spent: 10m
Work Description: tweise commented on pull request #8300: [BEAM-7035]
Compatible wire representation for timers in Python SDK
URL: https://github.com/apache/beam/pull/8300#discussion_r275350293
##
File path: sdks/python/apache_beam/runners/worker/bundle_processor.py
##
@@ -381,8 +381,12 @@ def set(self, ts):
windowed_value.WindowedValue(
(self._key, dict(timestamp=ts)), ts, (self._window,)))
- def clear(self, timestamp):
-self._receiver.receive((self._key, dict(clear=True)))
+ def clear(self):
+dummy_millis = int(common_urns.constants.MAX_TIMESTAMP_MILLIS.constant) + 1
Review comment:
Long.MAX_VALUE is `9223372036854775807`, which is different from
`MAX_TIMESTAMP_MILLIS`. The resulting timer timestamp is
`Timer{timestamp=294247-01-10T04:00:54.776Z, payload=[B@5aec2a63}`. (The code
in this PR is covered by a test, if there was an overflow the test would fail.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227662)
Time Spent: 2h 40m (was: 2.5h)
> Clear() method of OutputTimer is inconsistent
> -
>
> Key: BEAM-7035
> URL: https://issues.apache.org/jira/browse/BEAM-7035
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Reporter: Rakesh Kumar
>Assignee: Thomas Weise
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378]
> method of OutputTimer is not consistent ()
> The timestamp parameter is passed here but never used. Also in the [test
> cases
> |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]]
> and direct runner timer doesn't pass any parameter in the clear method
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227517&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227517 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 15/Apr/19 08:39 Start Date: 15/Apr/19 08:39 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#discussion_r275257192 ## File path: sdks/python/apache_beam/runners/worker/bundle_processor.py ## @@ -381,8 +381,12 @@ def set(self, ts): windowed_value.WindowedValue( (self._key, dict(timestamp=ts)), ts, (self._window,))) - def clear(self, timestamp): -self._receiver.receive((self._key, dict(clear=True))) + def clear(self): +dummy_millis = int(common_urns.constants.MAX_TIMESTAMP_MILLIS.constant) + 1 Review comment: Wouldn't this cause an overflow then? We cannot go past the maximum timestamp. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227517) Time Spent: 2.5h (was: 2h 20m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227390&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227390 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 14/Apr/19 20:39 Start Date: 14/Apr/19 20:39 Worklog Time Spent: 10m Work Description: tweise commented on issue #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#issuecomment-483056398 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227390) Time Spent: 2h 20m (was: 2h 10m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[
https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227381&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227381
]
ASF GitHub Bot logged work on BEAM-7035:
Author: ASF GitHub Bot
Created on: 14/Apr/19 18:14
Start Date: 14/Apr/19 18:14
Worklog Time Spent: 10m
Work Description: tweise commented on pull request #8300: [BEAM-7035]
Compatible wire representation for timers in Python SDK
URL: https://github.com/apache/beam/pull/8300#discussion_r275167931
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
##
@@ -378,14 +379,20 @@ public ByteBuffer getCurrentKey() {
private void setTimer(WindowedValue timerElement,
TimerInternals.TimerData timerData) {
try {
+ LOG.debug("Setting timer: {} {}", timerElement, timerData);
// KvToByteBufferKeySelector returns the key encoded
ByteBuffer encodedKey = (ByteBuffer) keySelector.getKey(timerElement);
// We have to synchronize to ensure the state backend is not
concurrently accessed by the
// state requests
try {
stateBackendLock.lock();
getKeyedStateBackend().setCurrentKey(encodedKey);
-timerInternals.setTimer(timerData);
+if
(timerData.getTimestamp().isAfter(GlobalWindow.TIMESTAMP_MAX_VALUE)) {
Review comment:
The constant is actually defined in BoundedWindow
https://github.com/apache/beam/blob/a9723d954a92391e9ca79130a199a8eaf38953f2/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/BoundedWindow.java#L57
I will change this to avoid confusion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227381)
Time Spent: 2h 10m (was: 2h)
> Clear() method of OutputTimer is inconsistent
> -
>
> Key: BEAM-7035
> URL: https://issues.apache.org/jira/browse/BEAM-7035
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Reporter: Rakesh Kumar
>Assignee: Thomas Weise
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378]
> method of OutputTimer is not consistent ()
> The timestamp parameter is passed here but never used. Also in the [test
> cases
> |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]]
> and direct runner timer doesn't pass any parameter in the clear method
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227380&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227380 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 14/Apr/19 18:08 Start Date: 14/Apr/19 18:08 Worklog Time Spent: 10m Work Description: tweise commented on pull request #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#discussion_r275167711 ## File path: sdks/python/apache_beam/runners/worker/bundle_processor.py ## @@ -381,8 +381,12 @@ def set(self, ts): windowed_value.WindowedValue( (self._key, dict(timestamp=ts)), ts, (self._window,))) - def clear(self, timestamp): -self._receiver.receive((self._key, dict(clear=True))) + def clear(self): +dummy_millis = int(common_urns.constants.MAX_TIMESTAMP_MILLIS.constant) + 1 Review comment: No, it refers to https://github.com/apache/beam/blob/7d6b7b89f5832d5d375059f9c1980181b443835b/model/pipeline/src/main/proto/beam_runner_api.proto#L40 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227380) Time Spent: 2h (was: 1h 50m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227373&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227373 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 14/Apr/19 17:44 Start Date: 14/Apr/19 17:44 Worklog Time Spent: 10m Work Description: tweise commented on pull request #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#discussion_r275166954 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -491,13 +491,29 @@ def estimate_size(self, value, nested=False): class TimestampCoderImpl(StreamCoderImpl): - """For internal use only; no backwards-compatibility guarantees.""" + """For internal use only; no backwards-compatibility guarantees. + + TODO: SDK agnostic encoding + For interoperability with Java SDK, encoding needs to match + that of the Java SDK InstantCoder. + https://github.com/apache/beam/blob/f5029b4f0dfff404310b2ef55e2632bbacc7b04f/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/InstantCoder.java#L79 + """ def encode_to_stream(self, value, out, nested): -out.write_bigendian_int64(value.micros) +millis = value.micros // 1000 Review comment: Yup, and timers only working because they would be set in the past (or fire on the final watermark). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227373) Time Spent: 1h 50m (was: 1h 40m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227371&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227371 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 14/Apr/19 17:43 Start Date: 14/Apr/19 17:43 Worklog Time Spent: 10m Work Description: tweise commented on pull request #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#discussion_r275166954 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -491,13 +491,29 @@ def estimate_size(self, value, nested=False): class TimestampCoderImpl(StreamCoderImpl): - """For internal use only; no backwards-compatibility guarantees.""" + """For internal use only; no backwards-compatibility guarantees. + + TODO: SDK agnostic encoding + For interoperability with Java SDK, encoding needs to match + that of the Java SDK InstantCoder. + https://github.com/apache/beam/blob/f5029b4f0dfff404310b2ef55e2632bbacc7b04f/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/InstantCoder.java#L79 + """ def encode_to_stream(self, value, out, nested): -out.write_bigendian_int64(value.micros) +millis = value.micros // 1000 Review comment: Yup, and timers only working because they would fire on the final watermark. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227371) Time Spent: 1h 40m (was: 1.5h) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[
https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227369&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227369
]
ASF GitHub Bot logged work on BEAM-7035:
Author: ASF GitHub Bot
Created on: 14/Apr/19 17:39
Start Date: 14/Apr/19 17:39
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8300: [BEAM-7035]
Compatible wire representation for timers in Python SDK
URL: https://github.com/apache/beam/pull/8300#discussion_r275166819
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##
@@ -1071,7 +1071,11 @@ public void deleteTimer(StateNamespace namespace,
String timerId) {
@Override
public void deleteTimer(StateNamespace namespace, String timerId,
TimeDomain timeDomain) {
- throw new UnsupportedOperationException("Canceling of a timer by ID is
not yet supported.");
+ try {
+cancelPendingTimerById(getContextTimerId(timerId, namespace));
Review comment:
Have to correct myself, I thought you were calling `cleanupPendingTimer`
here, but `cancelPendingTimerById` does delete the timer correctly. Sorry about
that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227369)
Time Spent: 1.5h (was: 1h 20m)
> Clear() method of OutputTimer is inconsistent
> -
>
> Key: BEAM-7035
> URL: https://issues.apache.org/jira/browse/BEAM-7035
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Reporter: Rakesh Kumar
>Assignee: Thomas Weise
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378]
> method of OutputTimer is not consistent ()
> The timestamp parameter is passed here but never used. Also in the [test
> cases
> |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]]
> and direct runner timer doesn't pass any parameter in the clear method
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[
https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227367&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227367
]
ASF GitHub Bot logged work on BEAM-7035:
Author: ASF GitHub Bot
Created on: 14/Apr/19 17:29
Start Date: 14/Apr/19 17:29
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8300: [BEAM-7035]
Compatible wire representation for timers in Python SDK
URL: https://github.com/apache/beam/pull/8300#discussion_r275165928
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
##
@@ -378,14 +379,20 @@ public ByteBuffer getCurrentKey() {
private void setTimer(WindowedValue timerElement,
TimerInternals.TimerData timerData) {
try {
+ LOG.debug("Setting timer: {} {}", timerElement, timerData);
// KvToByteBufferKeySelector returns the key encoded
ByteBuffer encodedKey = (ByteBuffer) keySelector.getKey(timerElement);
// We have to synchronize to ensure the state backend is not
concurrently accessed by the
// state requests
try {
stateBackendLock.lock();
getKeyedStateBackend().setCurrentKey(encodedKey);
-timerInternals.setTimer(timerData);
+if
(timerData.getTimestamp().isAfter(GlobalWindow.TIMESTAMP_MAX_VALUE)) {
Review comment:
Should this be `MAX_TIMESTAMP` (`Long.MAX_VALUE`)? There can be windows
between `GlobalWindow.MAX_TIMEMSTAMP` and `MAX_TIMESTAMP` because the former is
a standard day before the latter. Actually, `MAX_TIMESTAMP` would be good as a
value because triggers using it as a timestamp can never fire because the
Watermark cannot go past `MAX_TIMESTAMP`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227367)
Time Spent: 1h 20m (was: 1h 10m)
> Clear() method of OutputTimer is inconsistent
> -
>
> Key: BEAM-7035
> URL: https://issues.apache.org/jira/browse/BEAM-7035
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Reporter: Rakesh Kumar
>Assignee: Thomas Weise
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378]
> method of OutputTimer is not consistent ()
> The timestamp parameter is passed here but never used. Also in the [test
> cases
> |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]]
> and direct runner timer doesn't pass any parameter in the clear method
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227364&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227364 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 14/Apr/19 17:29 Start Date: 14/Apr/19 17:29 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#discussion_r275166287 ## File path: sdks/python/apache_beam/runners/worker/bundle_processor.py ## @@ -381,8 +381,12 @@ def set(self, ts): windowed_value.WindowedValue( (self._key, dict(timestamp=ts)), ts, (self._window,))) - def clear(self, timestamp): -self._receiver.receive((self._key, dict(clear=True))) + def clear(self): +dummy_millis = int(common_urns.constants.MAX_TIMESTAMP_MILLIS.constant) + 1 Review comment: Did you mean to use GlobalWindow.MAX_TIMESTAMP here? I think this should be just MAX_TIMESTAMP (see above). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227364) Time Spent: 1h (was: 50m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227365&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227365 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 14/Apr/19 17:29 Start Date: 14/Apr/19 17:29 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#discussion_r275166195 ## File path: sdks/python/apache_beam/coders/coder_impl.py ## @@ -491,13 +491,29 @@ def estimate_size(self, value, nested=False): class TimestampCoderImpl(StreamCoderImpl): - """For internal use only; no backwards-compatibility guarantees.""" + """For internal use only; no backwards-compatibility guarantees. + + TODO: SDK agnostic encoding + For interoperability with Java SDK, encoding needs to match + that of the Java SDK InstantCoder. + https://github.com/apache/beam/blob/f5029b4f0dfff404310b2ef55e2632bbacc7b04f/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/InstantCoder.java#L79 + """ def encode_to_stream(self, value, out, nested): -out.write_bigendian_int64(value.micros) +millis = value.micros // 1000 Review comment: Does that mean that timestamps were incorrect before? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227365) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[
https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227366&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227366
]
ASF GitHub Bot logged work on BEAM-7035:
Author: ASF GitHub Bot
Created on: 14/Apr/19 17:29
Start Date: 14/Apr/19 17:29
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8300: [BEAM-7035]
Compatible wire representation for timers in Python SDK
URL: https://github.com/apache/beam/pull/8300#discussion_r275166004
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##
@@ -1071,7 +1071,11 @@ public void deleteTimer(StateNamespace namespace,
String timerId) {
@Override
public void deleteTimer(StateNamespace namespace, String timerId,
TimeDomain timeDomain) {
- throw new UnsupportedOperationException("Canceling of a timer by ID is
not yet supported.");
+ try {
+cancelPendingTimerById(getContextTimerId(timerId, namespace));
Review comment:
This is not enough to clean the timer. The method call will only evict the
timer meta information used to delete timers by id (Flink does not allow to
delete timers by id, you always need to supply the original timestamp of the
the timer). See the deleteTimer method below for the additional delete step on
Flink's timer service.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227366)
Time Spent: 1h 10m (was: 1h)
> Clear() method of OutputTimer is inconsistent
> -
>
> Key: BEAM-7035
> URL: https://issues.apache.org/jira/browse/BEAM-7035
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Reporter: Rakesh Kumar
>Assignee: Thomas Weise
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378]
> method of OutputTimer is not consistent ()
> The timestamp parameter is passed here but never used. Also in the [test
> cases
> |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]]
> and direct runner timer doesn't pass any parameter in the clear method
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227357&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227357 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 14/Apr/19 16:14 Start Date: 14/Apr/19 16:14 Worklog Time Spent: 10m Work Description: tweise commented on issue #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300#issuecomment-483008383 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227357) Time Spent: 50m (was: 40m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Thomas Weise >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=227068&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227068 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 13/Apr/19 01:02 Start Date: 13/Apr/19 01:02 Worklog Time Spent: 10m Work Description: tweise commented on pull request #8300: [BEAM-7035] Compatible wire representation for timers in Python SDK URL: https://github.com/apache/beam/pull/8300 This change is just to fix the timer encoding. Changes to support cancellation of timers on the Flink runner side and test coverage to follow. Adding @lukecwik who will be familiar with the timestamp encoding workarounds :) Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- Pre-Commit Tests Status (on master branch) -
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=225239&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-225239 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 09/Apr/19 20:11 Start Date: 09/Apr/19 20:11 Worklog Time Spent: 10m Work Description: tweise commented on issue #8253: BEAM-7035: Remove unused parameter of clean method URL: https://github.com/apache/beam/pull/8253#issuecomment-481416355 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 225239) Time Spent: 0.5h (was: 20m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Rakesh Kumar >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=224657&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224657 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 08/Apr/19 23:06 Start Date: 08/Apr/19 23:06 Worklog Time Spent: 10m Work Description: rakeshcusat commented on issue #8253: BEAM-7035: Remove unused parameter of clean method URL: https://github.com/apache/beam/pull/8253#issuecomment-481039572 R: @tweise This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 224657) Time Spent: 20m (was: 10m) > Clear() method of OutputTimer is inconsistent > - > > Key: BEAM-7035 > URL: https://issues.apache.org/jira/browse/BEAM-7035 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Rakesh Kumar >Assignee: Rakesh Kumar >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > [Clear()|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/worker/bundle_processor.py#L378] > method of OutputTimer is not consistent () > The timestamp parameter is passed here but never used. Also in the [test > cases > |[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/userstate_test.py#L501]] > and direct runner timer doesn't pass any parameter in the clear method -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7035) Clear() method of OutputTimer is inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7035?focusedWorklogId=224656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-224656 ] ASF GitHub Bot logged work on BEAM-7035: Author: ASF GitHub Bot Created on: 08/Apr/19 23:01 Start Date: 08/Apr/19 23:01 Worklog Time Spent: 10m Work Description: rakeshcusat commented on pull request #8253: BEAM-7035: Remove unused parameter of clean method URL: https://github.com/apache/beam/pull/8253 Why? - Clean() method of OutputTimer is inconsistent. It takes the timestamp parameter and never use it. The test and direct runner version of clean method do not use extra parameter. **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- Pre-Commit Tests Statu
