[
https://issues.apache.org/jira/browse/BEAM-11105?focusedWorklogId=758132&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-758132
]
ASF GitHub Bot logged work on BEAM-11105:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Apr/22 21:40
Start Date: 18/Apr/22 21:40
Worklog Time Spent: 10m
Work Description: lostluck commented on code in PR #17374:
URL: https://github.com/apache/beam/pull/17374#discussion_r852433361
##########
sdks/go/pkg/beam/pardo.go:
##########
@@ -64,9 +66,16 @@ func TryParDo(s Scope, dofn interface{}, col PCollection,
opts ...Option) ([]PCo
}
var rc *coder.Coder
+ // Sdfs will always encode restrictions as KV<restriction, watermark
state | nil>
if fn.IsSplittable() {
sdf := (*graph.SplittableDoFn)(fn)
- rc, err = inferCoder(typex.New(sdf.RestrictionT()))
+ restT := typex.New(sdf.RestrictionT())
+ // If no watermark estimator state, use boolean as a placeholder
Review Comment:
Outside of specifying a new custom coder, the coder for GlobalWindows
encodes them 0 size, as it's a mono value type.
I suppose this related to my question during design, about whether we're
using the Watermark state all the time or not. A Watermark Estimator with no
state along with an SDF with type S should make the Restriction type S, rather
than trying to make up a type of WS to make KV<S, WS> valid.
Issue Time Tracking
-------------------
Worklog Id: (was: 758132)
Time Spent: 5h 10m (was: 5h)
> [Go SDK] Watermark Estimation
> -----------------------------
>
> Key: BEAM-11105
> URL: https://issues.apache.org/jira/browse/BEAM-11105
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Danny McCormick
> Priority: P3
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Allow self checkpointing SplittableDoFns to specify a watermark estimator.
> (To be updated once [https://github.com/apache/beam/pull/13160] is merged and
> the programming guide updated with SDF content.)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
