Elaborating on what Robert alluded to: when I wrote that runner author guide, portability was in its infancy. Now Beam Python can be run on Flink. So that guide is primarily focused on the "deserialize a Java DoFn and call its methods" approach. A decent amount of it is still really important to know, but is now the responsibility of the "SDK harness", aka language-specific coprocessor. For Python & Go & <insert new SDK language here> you really want to use the portability protos and the portable Flink runner is the best model.
Kenn On Fri, Feb 15, 2019 at 2:08 AM Robert Bradshaw <[email protected]> wrote: > On Fri, Feb 15, 2019 at 7:36 AM Can Gencer <[email protected]> wrote: > > > > We at Hazelcast are looking into writing a Beam runner for Hazelcast Jet > (https://github.com/hazelcast/hazelcast-jet). I wanted to introduce > myself as we'll likely have questions as we start development. > > Welcome! > > Hazelcast looks interesting, a Beam runner for it would be very cool. > > > Some of the things I'm wondering about currently: > > > > * Currently there seems to be a guide available at > https://beam.apache.org/contribute/runner-guide/ , is this up to date? Is > there anything in specific to be aware of when starting with a new runner > that's not covered here? > > That looks like a pretty good starting point. At a quick glance, I > don't see anything that looks out of date. Another resource that might > be helpful is a talk from last year on writing an SDK (but as it > mostly covers the runner-sdk interaction, it's also quite useful for > understanding the runner side: > > https://docs.google.com/presentation/d/1Cso0XP9dmj77OD9Bd53C1M3W1sPJF0ZnA20gzb2BPhE/edit#slide=id.p > And please feel free to ask any questions on this list as well; we'd > be happy to help. > > > * Should we be targeting the latest master which is at 2.12-SNAPSHOT or > a stable version? > > I would target the latest master. > > > * After a runner is developed, how is the maintenance typically handled, > as the runners seems to be part of Beam codebase? > > Either is possible. Several runner adapters are part of the Beam > codebase, but for example the IMB Streams Beam runner is not. There > are certainly pros and cons (certainly early on when the APIs > themselves were under heavy development it was easier to keep things > in sync in the same codebase, but things have mostly stabilized now). > A runner only becomes part of the Beam codebase if there are members > of the community committed to maintaining it (which could include > you). Both approaches are fine. > > - Robert >
