[
https://issues.apache.org/jira/browse/BEAM-10169?focusedWorklogId=445948&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445948
]
ASF GitHub Bot logged work on BEAM-10169:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 15/Jun/20 17:11
Start Date: 15/Jun/20 17:11
Worklog Time Spent: 10m
Work Description: lostluck commented on a change in pull request #11976:
URL: https://github.com/apache/beam/pull/11976#discussion_r440321837
##########
File path: sdks/go/pkg/beam/pardo_test.go
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package beam
+
+import (
+ "testing"
+)
+
+func TestRecommendParDo(t *testing.T) {
+ var tests = []struct {
+ name string
+ outputDim int
+ want string
+ }{
+ {"zero outputs", 0, "ParDo0"},
+ {"one output", 1, "ParDo"},
+ {"more than 7 outputs", 10, "ParDoN"},
+ }
+
+ for _, tt := range tests {
+ testName := tt.name
+ t.Run(testName, func(t *testing.T) {
+ got := recommendParDo(tt.outputDim)
+ if got != tt.want {
+ t.Errorf("RecommendParDo(%v) = %v, want %v",
tt.outputDim, got, tt.want)
+ }
+ })
+ }
+}
+
+func testFunction() int64 {
+ return 42
+}
+
+func TestFormatParDoError(t *testing.T) {
+ got := formatParDoError(testFunction, 2, 1)
Review comment:
Since the "complicated" bit for this is in the recommendParDo code,
which is already being tested very well, this test should be sufficient for the
parts in formatParDoError. I don't expect the formatting to change any time
soon.
We do have the english grammar nit ("1 outputs"), but handling that grammar
correction isn't worth the extra code to resolve it.
##########
File path: sdks/go/pkg/beam/pardo.go
##########
@@ -414,7 +412,36 @@ func ParDo6(s Scope, dofn interface{}, col PCollection,
opts ...Option) (PCollec
func ParDo7(s Scope, dofn interface{}, col PCollection, opts ...Option)
(PCollection, PCollection, PCollection, PCollection, PCollection, PCollection,
PCollection) {
ret := MustN(TryParDo(s, dofn, col, opts...))
if len(ret) != 7 {
- panic(fmt.Sprintf("expected 7 output. Found: %v", ret))
+ panic(formatParDoError(dofn, len(ret), 7))
}
return ret[0], ret[1], ret[2], ret[3], ret[4], ret[5], ret[6]
}
+
+//formatParDoError is a helper function to provide a more concise error
+// message to the users when a DoFn and its ParDo pairing is incorrect.
+//
+// We construct a new graph.Fn using the doFn which is passed. We explicitly
+// ignore the error since we already know that its already a DoFn type as
+// TryParDo would have panicked otherwise.
+func formatParDoError(doFn interface{}, emitSize int, parDoSize int) string {
+ doFun, _ := graph.NewFn(doFn)
+ doFnName := doFun.Name()
+
+ thisParDo := recommendParDo(parDoSize) // Conveniently keeps the API
slim.
Review comment:
Ah clever! But not in the hard to debug way. :D I like it.
Optional Nit: It's no longer really recommending, but naming the appropriate
DoFn for the size in this use. parDoForSize might be a more accurate name, but,
honestly, either is fine.
Had there only been the one call, I'd have suggested inlining it since the
switch isn't that large.
Either way, since it's being used twice, that justifies that switch being in
it's own function.
##########
File path: sdks/go/pkg/beam/pardo.go
##########
@@ -414,7 +412,36 @@ func ParDo6(s Scope, dofn interface{}, col PCollection,
opts ...Option) (PCollec
func ParDo7(s Scope, dofn interface{}, col PCollection, opts ...Option)
(PCollection, PCollection, PCollection, PCollection, PCollection, PCollection,
PCollection) {
ret := MustN(TryParDo(s, dofn, col, opts...))
if len(ret) != 7 {
- panic(fmt.Sprintf("expected 7 output. Found: %v", ret))
+ panic(formatParDoError(dofn, len(ret), 7))
}
return ret[0], ret[1], ret[2], ret[3], ret[4], ret[5], ret[6]
}
+
+//formatParDoError is a helper function to provide a more concise error
Review comment:
Nit: add a space between // and formatParDoError
##########
File path: sdks/go/pkg/beam/pardo_test.go
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package beam
+
+import (
+ "testing"
+)
+
+func TestRecommendParDo(t *testing.T) {
+ var tests = []struct {
+ name string
+ outputDim int
+ want string
+ }{
+ {"zero outputs", 0, "ParDo0"},
+ {"one output", 1, "ParDo"},
+ {"more than 7 outputs", 10, "ParDoN"},
+ }
+
+ for _, tt := range tests {
+ testName := tt.name
+ t.Run(testName, func(t *testing.T) {
+ got := recommendParDo(tt.outputDim)
+ if got != tt.want {
+ t.Errorf("RecommendParDo(%v) = %v, want %v",
tt.outputDim, got, tt.want)
+ }
+ })
+ }
+}
+
+func testFunction() int64 {
Review comment:
It might be worth commenting here that this function is just to validate
that the name is sourced properly in TestFormatParDoError since the number of
errors don't match.
##########
File path: sdks/go/pkg/beam/pardo_test.go
##########
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package beam
+
+import (
+ "testing"
+)
+
+func TestRecommendParDo(t *testing.T) {
+ var tests = []struct {
+ name string
+ outputDim int
+ want string
+ }{
+ {"zero outputs", 0, "ParDo0"},
+ {"one output", 1, "ParDo"},
Review comment:
Please add 3 more test cases here for 2 and 7,8 to cover those boundary
conditions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 445948)
Time Spent: 5h 20m (was: 5h 10m)
> ParDo* functions should declare the correct output N in their error message
> ---------------------------------------------------------------------------
>
> Key: BEAM-10169
> URL: https://issues.apache.org/jira/browse/BEAM-10169
> Project: Beam
> Issue Type: Improvement
> Components: sdk-go
> Reporter: Robert Burke
> Assignee: Aaron Tillekeratne
> Priority: P3
> Labels: noob, starter
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> User report noted the confusion in the error if you use a DoFn with 0 outputs
> with beam.ParDo instead of beam.ParDo0.
> In that case, a panic stack trace is followed by the cryptic: "expected 1
> output. Found: []"
> We can do better.
> While we can't change the return signature dynamically (that's for ParDoN
> only), we can instead clearly indicate:
> * the DoFn in question.
> * the number of outputs the DoFn has
> * and recommend using ParDo0, ParDo, ParDo2,...ParDo7, or ParDoN, as
> appropriate.
> https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/pardo.go#L361
> would need to change as well as any of the specific cases that follow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
