Thanks for the clarification, Pavel. Does it make sense adjust the code to overcome this inconvenience? I can try to work on draft-PR.
On Mon, May 8, 2023 at 8:30 AM Pavel Solomin <[email protected]> wrote: > Hello! > > What Beam aims to provide is keeping the same code of the pipeline itself > when you switch from one runner to another. But Options is runner-specific. > > When you change runners, you still need to rebuild the project with the > correct runner and runner options. For example: > > > https://beam.apache.org/releases/javadoc/current/org/apache/beam/runners/flink/FlinkPipelineOptions.html > > > https://beam.apache.org/releases/javadoc/current/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html > > In my Beam projects I was managing that via maven profiles: > > - pipeline code is packed into its own module, does not channge when I > switch runners, and is always included into the final artifact > > - pipeline options and Main have very little code (only calls generic > pipeline code), are runner-specific, and I compile them based on the > profile selected -P flinkRunner / -P dataflowRunner / etc > > > On Sunday, 7 May 2023, Mikhail Khludnev <[email protected]> wrote: > > Hello, > > I'm starting to learn Beam. I took Java SDK and play with > WordCount/Starter. > > I'm able to run pipeline in embedded Flink via --runner=FlinkRunner > > INFO: Starting Flink Mini Cluster > > However passing --flinkMaster=localhost:8081 doesn't really work due to > > java.lang.IllegalArgumentException: Class > interface WordCountOptions missing a property named 'flinkMaster'. > > It somewhat reasonable because > > interface WordCountOptions extends PipelineOptions { > > I can call withoutStrictParsing() that just bypasses check, but doesn't > pass flinkMaster argument to FlinkRunner and I've got Flink Mini Cluster > again. > > Another way is to extends my custom options from FlinkPipelineOptions it > makes it work on remote cluster. Fine. > > Here I'm missing something, it seems like I can't run this pipeline by > another runner, let's say Dataflow without changing the code. > > How can I build a runner agnostic pipeline? > > Sorry for asking a dummy question. > > > > -- > > Sincerely yours > > Mikhail Khludnev > > > > -- > Best Regards, > Pavel Solomin > > Tel: +351 962 950 692 | Skype: pavel_solomin | Linkedin > <https://www.linkedin.com/in/pavelsolomin> > > > > > -- Sincerely yours Mikhail Khludnev https://t.me/MUST_SEARCH A caveat: Cyrillic!
