[jira] [Updated] (BEAM-2535) Allow explicit output time independent of firing specification for all timers
[ https://issues.apache.org/jira/browse/BEAM-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Batkhuyag Batsaikhan updated BEAM-2535: --- Description: Today, we have insufficient control over the event time timestamp of elements output from a timer callback. 1. For an event time timer, it is the timestamp of the timer itself. 2. For a processing time timer, it is the current input watermark at the time of processing. But for both of these, we may want to reserve the right to output a particular time, aka set a "watermark hold". A naive implementation of a {{TimerWithWatermarkHold}} would work for making sure output is not droppable, but does not fully explain window expiration and late data/timer dropping. In the natural interpretation of a timer as a feedback loop on a transform, timers should be viewed as another channel of input, with a watermark, and items on that channel _all need event time timestamps even if they are delivered according to a different time domain_. I propose that the specification for when a timer should fire should be separated (with nice defaults) from the specification of the event time of resulting outputs. These timestamps will determine a side channel with a new "timer watermark" that constrains the output watermark. - We still need to fire event time timers according to the input watermark, so that event time timers fire. - Late data dropping and window expiration will be in terms of the minimum of the input watermark and the timer watermark. In this way, whenever a timer is set, the window is not going to be garbage collected. - We will need to make sure we have a way to "wake up" a window once it is expired; this may be as simple as exhausting the timer channel as soon as the input watermark indicates expiration of a window This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It seems reasonable to use timers as an implementation detail (e.g. in runners-core utilities) without wanting any of this additional machinery. For example, if there is no possibility of output from the timer callback. was: Today, we have insufficient control over the event time timestamp of elements output from a timer callback. 1. For an event time timer, it is the timestamp of the timer itself. 2. For a processing time timer, it is the current input watermark at the time of processing. But for both of these, we may want to reserve the right to output a particular time, aka set a "watermark hold". A naive implementation of a {{TimerWithWatermarkHold}} would work for making sure output is not droppable, but does not fully explain window expiration and late data/timer dropping. In the natural interpretation of a timer as a feedback loop on a transform, timers should be viewed as another channel of input, with a watermark, and items on that channel _all need event time timestamps even if they are delivered according to a different time domain_. I propose that the specification for when a timer should fire should be separated (with nice defaults) from the specification of the event time of resulting outputs. These timestamps will determine a side channel with a new "timer watermark" that constrains the output watermark. - We still need to fire event time timers according to the input watermark, so that event time timers fire. - Late data dropping and window expiration will be in terms of the minimum of the input watermark and the timer watermark. In this way, whenever a timer is set, the window is not going to be garbage collected. - We will need to make sure we have a way to "wake up" a window once it is expired; this may be as simple as exhausting the timer channel as soon as the input watermark indicates expiration of a window This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It seems reasonable to use timers as an implementation detail (e.g. in runners-core utilities) without wanting any of this additional machinery. For example, if there is no possibility of output from the timer callback. > Allow explicit output time independent of firing specification for all timers > - > > Key: BEAM-2535 > URL: https://issues.apache.org/jira/browse/BEAM-2535 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-java-core >Reporter: Kenneth Knowles >Assignee: Batkhuyag Batsaikhan >Priority: Major > > Today, we have insufficient control over the event time timestamp of elements > output from a timer callback. > 1. For an event time timer, it is the timestamp of the timer itself. > 2. For a processing time timer, it is the current input watermark at the > time of processing. > But for both of these, we may want to reserve the right to output a > part
[jira] [Updated] (BEAM-2535) Allow explicit output time independent of firing specification for all timers
[ https://issues.apache.org/jira/browse/BEAM-2535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-2535: -- Description: Today, we have insufficient control over the event time timestamp of elements output from a timer callback. 1. For an event time timer, it is the timestamp of the timer itself. 2. For a processing time timer, it is the current input watermark at the time of processing. But for both of these, we may want to reserve the right to output a particular time, aka set a "watermark hold". A naive implementation of a {{TimerWithWatermarkHold}} would work for making sure output is not droppable, but does not fully explain window expiration and late data/timer dropping. In the natural interpretation of a timer as a feedback loop on a transform, timers should be viewed as another channel of input, with a watermark, and items on that channel _all need event time timestamps even if they are delivered according to a different time domain_. I propose that the specification for when a timer should fire should be separated (with nice defaults) from the specification of the event time of resulting outputs. These timestamps will determine a side channel with a new "timer watermark" that constrains the output watermark. - We still need to fire event time timers according to the input watermark, so that event time timers fire. - Late data dropping and window expiration will be in terms of the minimum of the input watermark and the timer watermark. In this way, whenever a timer is set, the window is not going to be garbage collected. - We will need to make sure we have a way to "wake up" a window once it is expired; this may be as simple as exhausting the timer channel as soon as the input watermark indicates expiration of a window This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It seems reasonable to use timers as an implementation detail (e.g. in runners-core utilities) without wanting any of this additional machinery. For example, if there is no possibility of output from the timer callback. was: Today, we have insufficient control over the event time timestamp of elements output from a timer callback. 1. For an event time timer, it is the timestamp of the timer itself. 2. For a processing time timer, it is the current input watermark at the time of processing. But for both of these, we may want to reserve the right to output a particular time, aka set a "watermark hold". A naive implementation of a {{TimerWithWatermarkHold}} would work for making sure output is not droppable, but does not fully explain window expiration and late data/timer dropping. In the natural interpretation of a timer as a feedback loop on a transform, timers should be viewed as another channel of input, with a watermark, and items on that channel _all need event time timestamps even if they are delivered according to a different time domain_. I propose that the specification for when a timer should fire should be separated (with nice defaults) from the specification of the event time of resulting outputs. These timestamps will determine a side channel with a new "timer watermark" that constrains the output watermark. - We still need to fire event time timers according to the input watermark, so that event time timers fire. - Late data dropping and window expiration will be in terms of the minimum of the input watermark and the timer watermark. In this way, whenever a timer is set, the window is not going to be garbage collected. - We will need to make sure we have a way to "wake up" a window once it is expired; this may be as simple as exhausting the timer channel as soon as the input watermark indicates expiration of a window > Allow explicit output time independent of firing specification for all timers > - > > Key: BEAM-2535 > URL: https://issues.apache.org/jira/browse/BEAM-2535 > Project: Beam > Issue Type: New Feature > Components: beam-model, sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles > > Today, we have insufficient control over the event time timestamp of elements > output from a timer callback. > 1. For an event time timer, it is the timestamp of the timer itself. > 2. For a processing time timer, it is the current input watermark at the time > of processing. > But for both of these, we may want to reserve the right to output a > particular time, aka set a "watermark hold". > A naive implementation of a {{TimerWithWatermarkHold}} would work for making > sure output is not droppable, but does not fully explain window expiration > and late data/timer dropping. > In the natural interpretation of a timer as a feedback loop on a transform, > timers should be viewed as a