[
https://issues.apache.org/jira/browse/BEAM-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327724#comment-16327724
]
Eugene Kirpichov commented on BEAM-71:
--------------------------------------
As far as SDF is concerned, the only relevant method is
DoFn.ProcessContext.updateWatermark()
[https://github.com/apache/beam/blob/d77f3ec00e20a93fb8cbc4b790603e19ce03568a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L301]
This issue should probably be implemented as a utility class that people can
use in arbitrary ways, both from an UnboundedSource and from an SDF. E.g. an
UnboundedReader.getWatermark() could delegate to utility.getWatermark(), and an
SDF could call c.updateWatermark(utility.getWatermark()).
Here's another interesting example of computing watermarks (in an SDF):
[https://github.com/apache/beam/blob/d77f3ec00e20a93fb8cbc4b790603e19ce03568a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Watch.java]
- search for "watermark", it's pretty fancy, though it might be too fancy to
consider it as something that the current issue should cover.
> Watermark library
> -----------------
>
> Key: BEAM-71
> URL: https://issues.apache.org/jira/browse/BEAM-71
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
> Reporter: Daniel Halperin
> Assignee: Tarush Grover
> Priority: Major
> Labels: Watermark
>
> See also: BEAM-70
> We should have a library of watermark implementations:
> * now: for when items use arrival timestamps
> * latest element: for when events arrive in timestamp order
> * moving average over last K time intervals
> * bucketed moving average, ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
