You can always do it via System, populating the options, right ?
Regards JB On 05/20/2016 11:07 AM, Ismaël Mejía wrote:
Hello again, I just executed our pipeline with the changes you just merged, and guess what ? It works perfectly (in batch mode), amazing job guys. I have just one extra question, is there a way to configure the pipeline options with system properties (-D...). e.g. to configure the streaming/batch mode and avoid to do this explicitly in the code (to remove the explicit FlinkPipelineOptions import) ? Ismaël On Fri, May 20, 2016 at 10:06 AM, Maximilian Michels <[email protected] <mailto:[email protected]>> wrote: Hi Ismael, Side inputs are not supported when the Flink Runner runs in streaming mode. Actually, we just merged changes which allow you to use the batch mode (streaming set to false) with Windows and side inputs. So just checkout the latest Beam master and rebuild. Set the streaming flag to false again and rerun your example. Thanks, Max On Fri, May 20, 2016 at 9:22 AM, Ismaël Mejía <[email protected] <mailto:[email protected]>> wrote: > > Hello, > > Thanks for your answer, I am really happy to know you are working on > removing > this flag for batch/streaming since this is not really Beam-like. > > I tried to execute the pipeline again in streaming mode, and I got this > exception now: > > java.lang.UnsupportedOperationException: The transform > View.CreatePCollectionView is currently not supported. > > Any info about this one ? > > Thanks again, > Ismael > > > On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek <[email protected] <mailto:[email protected]>> > wrote: >> >> Hi, >> as an addition, there are plans to get rid of the "streaming" flag for all >> runners and determine the mode from the job. >> >> -Aljoscha >> >> On Thu, 19 May 2016 at 18:52 Maximilian Michels <[email protected] <mailto:[email protected]>> wrote: >>> >>> Hi Ismaël, >>> >>> There is a pull request which enables Windows on batch. It will be >>> merged soon. In the meantime, please use the streaming execution mode >>> of the Flink Runner. >>> >>> Use setStreaming(true) to enable it on the FlinkPipelineOptions. >>> >>> Cheers, >>> Max >>> >>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <[email protected] <mailto:[email protected]>> wrote: >>> > I was trying to run a local Pipeline we have (basically a batch one we >>> > rewrote from TPC-H) in all the possible runners, for the moment I had >>> > success with the DirectPipelineRunner, InProcessPipelineRunner and >>> > SparkPipelineRunner. However when I executed it in the >>> > FlinkPipelineRunner I >>> > got this exception: >>> > >>> > java.lang.UnsupportedOperationException: The transform Window.Into() is >>> > currently not supported. >>> > >>> > I thought it was probably a configuration error since I remember that >>> > in the >>> > flink runner we had to explicitly define if it was a batch or streaming >>> > job, >>> > but when I checked the options: >>> > >>> > Current Settings: >>> > appName: TestQ08Task >>> > filesToStage: ... >>> > flinkMaster: [auto] >>> > parallelism: 1 >>> > runner: class org.apache.beam.runners.flink.FlinkPipelineRunner >>> > stableUniqueNames: WARNING >>> > streaming: false >>> > >>> > I noticed I was in batch mode, is there something missing in the >>> > options >>> > configuration, or is it that the UnsoppertedOperation is a WIP or not >>> > supported at all by Flink ? >>> > >>> > And extra question, this option to define if it is in batch or stream >>> > mode >>> > is going to stay for long, can't be this inferred some how ? >>> > >>> > Thanks, >>> > -Ismaël >>> > > >
-- Jean-Baptiste Onofré [email protected] http://blog.nanthrax.net Talend - http://www.talend.com
