> On 23 Jun 2023, at 17:40, Robert Bradshaw via user <user@beam.apache.org> > wrote: > > On Fri, Jun 23, 2023, 7:37 AM Alexey Romanenko <aromanenko....@gmail.com > <mailto:aromanenko....@gmail.com>> wrote: >> If Beam Runner Authoring Guide is rather high-level for you, then, at fist, >> I’d suggest to answer two questions for yourself: >> - Am I going to implement a portable runner or native one? > > > The answer to this should be portable, as non-portable ones will be > deprecated.
Well, actually this is a question that I don’t remember we discussed here in details before and had a common agreement. Actually, I’m not sure that I understand clearly what is meant by “deprecation" in this case. For example, Portable Spark Runner is heavily actually based on native Spark RDD runner and its translations. So, which part should be deprecated and what is a reason for that? Well, anyway I guess it’s off topic here. Also, we don’t know if this new runner will be contributed back to Beam, what is a runtime and what actually is a final goal of it. So I agree that more details on this would be useful. — Alexey > >> - Which SDK I should use for this runner? > > > The answer to the above question makes this one moot :). > > On a more serious note, could you tell us a bit more about the runner you're > looking at implementing? > >> >> Then, depending on answers, I’d suggest to take as an example one of the >> most similar Beam runners and use it as a more detailed source of >> information along with Beam runner doc mentioned before. >> >> — >> Alexey >> >>> On 22 Jun 2023, at 14:39, Joey Tran <joey.t...@schrodinger.com >>> <mailto:joey.t...@schrodinger.com>> wrote: >>> >>> Hi Beam community! >>> >>> I'm interested in trying to implement a runner with my company's execution >>> environment but I'm struggling to get started. I've read the docs page >>> <https://beam.apache.org/contribute/runner-guide/#testing-your-runner> on >>> implementing a runner but it's quite high level. Anyone have any concrete >>> suggestions on getting started? >>> >>> I've started by cloning and running the hello world example >>> <https://github.com/apache/beam-starter-python>. I've then subclassed >>> `PipelineRunner >>> <https://github.com/apache/beam/blob/9d0fc05d0042c2bb75ded511497e1def8c218c33/sdks/python/apache_beam/runners/runner.py#L103>` >>> to create my own custom runner but at this point I'm a bit stuck. My >>> custom runner just looks like >>> >>> class CustomRunner(runner.PipelineRunner): >>> def run_pipeline(self, pipeline, >>> options): >>> self.visit_transforms(pipeline, options) >>> >>> And when using it I get an error about not having implemented "Impulse" >>> >>> NotImplementedError: Execution of [<Impulse(PTransform) label=[Impulse]>] >>> not implemented in runner <my_app.app.CustomRunner object at 0x135d9ff40>. >>> >>> Am I going about this the right way? Are there tests I can run my custom >>> runner against to validate it beyond just running the hello world example? >>> I'm finding myself just digging through the beam source to try to piece >>> together how a runner works and I'm struggling to get a foothold. Any >>> guidance would be greatly appreciated, especially if anyone has any >>> experience implementing their own python runner. >>> >>> Thanks in advance! Also, could I get a Slack invite? >>> Cheers, >>> Joey >>
