Hi, you should be able to set the runner using command line options, e.g: --runner=org.apache.beam.runners.flink.FlinkPipelineRunner --streaming=false --parallelism=8
And in the program have something like: Options options = PipelineOptionsFactory.fromArgs(args).withValidation(); -Aljoscha On Fri, 20 May 2016 at 13:59 Jean-Baptiste Onofré <[email protected]> wrote: > You can always do it via System, populating the options, right ? > > Regards > JB > > On 05/20/2016 11:07 AM, Ismaël Mejía wrote: > > Hello again, > > > > I just executed our pipeline with the changes you just merged, and guess > > what ? It works perfectly (in batch mode), amazing job guys. I have just > one > > extra question, is there a way to configure the pipeline options with > system > > properties (-D...). e.g. to configure the streaming/batch mode and > > avoid to do > > this explicitly in the code (to remove the explicit FlinkPipelineOptions > > import) ? > > > > Ismaël > > > > > > On Fri, May 20, 2016 at 10:06 AM, Maximilian Michels <[email protected] > > <mailto:[email protected]>> wrote: > > > > Hi Ismael, > > > > Side inputs are not supported when the Flink Runner runs in streaming > > mode. Actually, we just merged changes which allow you to use the > > batch mode (streaming set to false) with Windows and side inputs. > > > > So just checkout the latest Beam master and rebuild. Set the > streaming > > flag to false again and rerun your example. > > > > Thanks, > > Max > > > > On Fri, May 20, 2016 at 9:22 AM, Ismaël Mejía <[email protected] > > <mailto:[email protected]>> wrote: > > > > > > Hello, > > > > > > Thanks for your answer, I am really happy to know you are working > on > > > removing > > > this flag for batch/streaming since this is not really Beam-like. > > > > > > I tried to execute the pipeline again in streaming mode, and I > > got this > > > exception now: > > > > > > java.lang.UnsupportedOperationException: The transform > > > View.CreatePCollectionView is currently not supported. > > > > > > Any info about this one ? > > > > > > Thanks again, > > > Ismael > > > > > > > > > On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek > > <[email protected] <mailto:[email protected]>> > > > wrote: > > >> > > >> Hi, > > >> as an addition, there are plans to get rid of the "streaming" > > flag for all > > >> runners and determine the mode from the job. > > >> > > >> -Aljoscha > > >> > > >> On Thu, 19 May 2016 at 18:52 Maximilian Michels <[email protected] > > <mailto:[email protected]>> wrote: > > >>> > > >>> Hi Ismaël, > > >>> > > >>> There is a pull request which enables Windows on batch. It will > be > > >>> merged soon. In the meantime, please use the streaming > > execution mode > > >>> of the Flink Runner. > > >>> > > >>> Use setStreaming(true) to enable it on the FlinkPipelineOptions. > > >>> > > >>> Cheers, > > >>> Max > > >>> > > >>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía > > <[email protected] <mailto:[email protected]>> wrote: > > >>> > I was trying to run a local Pipeline we have (basically a > > batch one we > > >>> > rewrote from TPC-H) in all the possible runners, for the > > moment I had > > >>> > success with the DirectPipelineRunner, > > InProcessPipelineRunner and > > >>> > SparkPipelineRunner. However when I executed it in the > > >>> > FlinkPipelineRunner I > > >>> > got this exception: > > >>> > > > >>> > java.lang.UnsupportedOperationException: The transform > > Window.Into() is > > >>> > currently not supported. > > >>> > > > >>> > I thought it was probably a configuration error since I > > remember that > > >>> > in the > > >>> > flink runner we had to explicitly define if it was a batch or > > streaming > > >>> > job, > > >>> > but when I checked the options: > > >>> > > > >>> > Current Settings: > > >>> > appName: TestQ08Task > > >>> > filesToStage: ... > > >>> > flinkMaster: [auto] > > >>> > parallelism: 1 > > >>> > runner: class > org.apache.beam.runners.flink.FlinkPipelineRunner > > >>> > stableUniqueNames: WARNING > > >>> > streaming: false > > >>> > > > >>> > I noticed I was in batch mode, is there something missing in > the > > >>> > options > > >>> > configuration, or is it that the UnsoppertedOperation is a > > WIP or not > > >>> > supported at all by Flink ? > > >>> > > > >>> > And extra question, this option to define if it is in batch > > or stream > > >>> > mode > > >>> > is going to stay for long, can't be this inferred some how ? > > >>> > > > >>> > Thanks, > > >>> > -Ismaël > > >>> > > > > > > > > > > > > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
