Hi, a quick related question: In the Flink runner we basically see everything as one big bundle, i.e. we call startBundle() once at the beginning and then keep processing indefinitely, never calling finishBundle(). Is this also correct behavior?
Best, Aljoscha On Tue, 7 Jun 2016 at 20:44 Thomas Groh <tg...@google.com.invalid> wrote: > Hey everyone; > > I'm starting to work on BEAM-38 ( > https://issues.apache.org/jira/browse/BEAM-38), which enables an > optimization for runners with many small bundles. BEAM-38 allows runners to > reuse DoFn instances so long as that DoFn has not terminated abnormally. > This replaces the previous requirement that a DoFn be used for only a > single bundle if either of startBundle or finishBundle have been > overwritten. > > DoFn deserialization-per-bundle can be a significance performance > bottleneck when there are many small bundles, as is common in streaming > executions. It has also surfaced as the cause of much of the current > slowness in the new InProcessRunner. > > Existing Runners do not require any changes; they may choose to take > advantage of of the new optimization opportunity. However, user DoFns may > need to be revised to properly set up and tear down state in startBundle > and finishBundle, respectively, if the depended on only being used for a > single bundle. > > The first two updates are already in pull requests: > > PR #419 (https://github.com/apache/incubator-beam/pull/419) updates the > Javadoc to the new spec > PR #418 (https://github.com/apache/incubator-beam/pull/418) updates the > DirectRunner to reuse DoFns according to the new policy. > > Yours, > > Thomas >
