Combine admits many more execution plans than stateful ParDo: - "Combiner lifting" or "mapper-side combine", in which the CombineFn is used to reduce data before shuffling. This is tremendous in batch, but can still matter in streaming. - Hot key fanout & recombine. This is important in both batch & streaming.
I tried to cover the issues a little in this section of my blog post on state, because it also answers the converse question: why/when would you use state (without timers) when Combine is so similar? https://beam.apache.org/blog/2017/02/13/stateful-processing.html#how-does-stateful-processing-fit-into-the-beam-model And here's a slide with the same idea but side-by-side illustrations: https://s.apache.org/ffsf-2017-beam-state#slide=id.g1dbf0d46d2_0_258 Kenn On Tue, Mar 12, 2019 at 6:55 AM Steve Niemitz <[email protected]> wrote: > Hi all. > > I'm curious if anyone has done any comparison of the performance of a > pipeline that uses CombineByKey, vs one that uses a stateful DoFn with > combining state. [1] > > More specifically, if I had a pipeline that had a CombineByKey configured > with early firings every N minutes, and I replaced the CBK with a stateful > DoFn with combining state and a timer that fired every N minutes instead, > would there be a (significant?) performance difference? Specifically I'm > using dataflow (with streaming engine) but I'd be curious for other runners > as well > > If no one has tried this I might do a benchmark to test, I'd be very > interested to see the results. > > [1] > https://beam.apache.org/releases/javadoc/2.11.0/org/apache/beam/sdk/state/CombiningState.html >
