Your reasoning is correct around the withHotkeyFanout hint and it is to help runners know that there is likely one or more keys that will have significantly more data then the others but the logic around how it is broken up is runner dependent and whether they rely on the hint or not is also runner dependent. If a runner was smart enough, it wouldn't need the hint and could automatically detect hotkeys and do the right thing. I would take a look at this doc[1] to learn about how the optimization can work from a runners perspective. Some runners never perform the PreCombine, while others may have multiple rounds of it but the most common case is that there is only a single PreCombine (assuming it is allowed).
1: https://s.apache.org/beam-runner-api-combine-model On Tue, Jun 30, 2020 at 10:56 AM Julien Phalip <jpha...@gmail.com> wrote: > Hi, > > I had a question about how combiners work, particularly on how the > combined PCollection's subsets are initially formed. > > I understand that, according to the documentation > <https://beam.apache.org/documentation/programming-guide/#combine>, a > combiner allows parallelizing the computation to multiple workers by > breaking up the PCollection into subsets. I like the database analogy given > in this post > <https://cloud.google.com/blog/products/gcp/writing-dataflow-pipelines-with-scalability-in-mind>, > which says that it is similar to pushing down a predicate. > > I also understand that it is possible to use withFanout or > withHotkeyFanout to provide some explicit logic as a hint on how to > manage the distribution. > > What is unclear to me, however, is whether by default the runner already > plans the distribution of the computation, even when no explicit hints are > provided. I'm guessing perhaps it always breaks up the PCollection into > bundles > <https://beam.apache.org/documentation/runtime/model/#bundling-and-persistence> > (similar to DoFns), then the combiner runs the combination on each bundle, > saves the result into intermediary accumulators, and those results then > bubble up recursively to the top? If that's the case, then I assume that > the purpose of withFanout and withHotKeyFanout is to further break up > those initially pre-created bundles into even smaller subsets? Or am I > guessing this wrong? :) > > I couldn't find a clear description in the documentation on how the > PCollection subsets are initially formed. Please let me know if you have > some details on that, or if it is already documented somewhere. > > Thank you! > > Julien >
