youngoli commented on a change in pull request #11197: [BEAM-8292] Portable
Reshuffle for Go SDK
URL: https://github.com/apache/beam/pull/11197#discussion_r396887810
##########
File path: sdks/go/pkg/beam/gbk.go
##########
@@ -95,3 +95,52 @@ func TryCoGroupByKey(s Scope, cols ...PCollection)
(PCollection, error) {
ret.SetCoder(NewCoder(ret.Type()))
return ret, nil
}
+
+// Reshuffle copies a PCollection of the same kind and using the same element
+// coder, and maintains the same windowing information. Importantly, it allows
+// the result PCollection to be processed with a different sharding, in a
+// different stage than the input PCollection.
+//
+// For example, if a computation needs a lot of parallelism but
+// produces only a small amount of output data, then the computation
+// producing the data can run with as much parallelism as needed,
+// while the output file is written with a smaller amount of
+// parallelism, using the following pattern:
+//
+// pc := bigHairyComputationNeedingParallelism(scope) // PCollection<string>
+// resharded := beam.Reshard(scope, pc) // PCollection<string>
Review comment:
Here and elsewhere in this comment, Reshuffle is referred to as "Reshard". I
think it's fine to refer to it as a reshard informally, since that's what it
functionally is, but the places where it's used as a proper noun should be
switched to Reshuffle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
