[
https://issues.apache.org/jira/browse/BEAM-3301?focusedWorklogId=395343&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395343
]
ASF GitHub Bot logged work on BEAM-3301:
----------------------------------------
Author: ASF GitHub Bot
Created on: 29/Feb/20 00:03
Start Date: 29/Feb/20 00:03
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #10991: [BEAM-3301]
Refactor DoFn validation & allow specifying main inputs.
URL: https://github.com/apache/beam/pull/10991#discussion_r385468257
##########
File path: sdks/go/pkg/beam/core/graph/fn.go
##########
@@ -209,21 +209,58 @@ func (f *DoFn) RestrictionT() *reflect.Type {
// a KV or not based on the other signatures (unless we're more loose about
which
// sideinputs are present). Bind should respect that.
+// Constants so we can avoid magic numbers in validation. Represent number of
+// DoFn main inputs based on what kind of input the DoFn has.
+const (
+ unknownInNum = -1 // Used when we don't know the number of main inputs.
+ singleInNum = 1
+ kvInNum = 2
+)
+
// NewDoFn constructs a DoFn from the given value, if possible.
func NewDoFn(fn interface{}) (*DoFn, error) {
ret, err := NewFn(fn)
if err != nil {
return nil, errors.WithContext(errors.Wrapf(err, "invalid
DoFn"), "constructing DoFn")
}
- return AsDoFn(ret)
+ return AsDoFn(ret, unknownInNum)
}
-// AsDoFn converts a Fn to a DoFn, if possible.
-func AsDoFn(fn *Fn) (*DoFn, error) {
+// NewDoFnKv constructs a DoFn from the given value, if possible, with
+// improved validation from knowing whether the DoFn's main input is a KV or
+// single element.
+func NewDoFnKv(fn interface{}, mainKv bool) (*DoFn, error) {
Review comment:
With the name NewDoFnKv, it sounds like it's already assuming that a DoFn KV
is being passed in. It's OK for there to be special purpose methods that only
do one thing.
Another option to consider instead of having two (or N) methods, consider
extending the current NewDoFn with a variadic an Option type. (eg. opts
...Option), this lets existing callers keep things the same, but allow for
expanding things in the future. Option should probably be either a function
type, or an interface type with private methods, and the options are provided
by other methods in the package. eg. graph.NewDoFn(fn, graph.HasKVInput(),
graph.HasRestriction()). This is valuable if we think being able to expand
things in the future, but also lets us mix and match more easily later on. This
way we can keep the existing behavior when there are no options, but keep the
documentation of all the various uses in one place on the NewDoFn method
referring to the option returning methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 395343)
Time Spent: 2h (was: 1h 50m)
> Go SplittableDoFn support
> -------------------------
>
> Key: BEAM-3301
> URL: https://issues.apache.org/jira/browse/BEAM-3301
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Henning Rohde
> Assignee: Daniel Oliveira
> Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> SDFs will be the only way to add streaming and liquid sharded IO for Go.
> Design doc: https://s.apache.org/splittable-do-fn
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
