Hello again, I just executed our pipeline with the changes you just merged, and guess what ? It works perfectly (in batch mode), amazing job guys. I have just one extra question, is there a way to configure the pipeline options with system properties (-D...). e.g. to configure the streaming/batch mode and avoid to do this explicitly in the code (to remove the explicit FlinkPipelineOptions import) ?
Ismaël On Fri, May 20, 2016 at 10:06 AM, Maximilian Michels <[email protected]> wrote: > Hi Ismael, > > Side inputs are not supported when the Flink Runner runs in streaming > mode. Actually, we just merged changes which allow you to use the > batch mode (streaming set to false) with Windows and side inputs. > > So just checkout the latest Beam master and rebuild. Set the streaming > flag to false again and rerun your example. > > Thanks, > Max > > On Fri, May 20, 2016 at 9:22 AM, Ismaël Mejía <[email protected]> wrote: > > > > Hello, > > > > Thanks for your answer, I am really happy to know you are working on > > removing > > this flag for batch/streaming since this is not really Beam-like. > > > > I tried to execute the pipeline again in streaming mode, and I got this > > exception now: > > > > java.lang.UnsupportedOperationException: The transform > > View.CreatePCollectionView is currently not supported. > > > > Any info about this one ? > > > > Thanks again, > > Ismael > > > > > > On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek <[email protected]> > > wrote: > >> > >> Hi, > >> as an addition, there are plans to get rid of the "streaming" flag for > all > >> runners and determine the mode from the job. > >> > >> -Aljoscha > >> > >> On Thu, 19 May 2016 at 18:52 Maximilian Michels <[email protected]> wrote: > >>> > >>> Hi Ismaël, > >>> > >>> There is a pull request which enables Windows on batch. It will be > >>> merged soon. In the meantime, please use the streaming execution mode > >>> of the Flink Runner. > >>> > >>> Use setStreaming(true) to enable it on the FlinkPipelineOptions. > >>> > >>> Cheers, > >>> Max > >>> > >>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <[email protected]> > wrote: > >>> > I was trying to run a local Pipeline we have (basically a batch one > we > >>> > rewrote from TPC-H) in all the possible runners, for the moment I > had > >>> > success with the DirectPipelineRunner, InProcessPipelineRunner and > >>> > SparkPipelineRunner. However when I executed it in the > >>> > FlinkPipelineRunner I > >>> > got this exception: > >>> > > >>> > java.lang.UnsupportedOperationException: The transform Window.Into() > is > >>> > currently not supported. > >>> > > >>> > I thought it was probably a configuration error since I remember that > >>> > in the > >>> > flink runner we had to explicitly define if it was a batch or > streaming > >>> > job, > >>> > but when I checked the options: > >>> > > >>> > Current Settings: > >>> > appName: TestQ08Task > >>> > filesToStage: ... > >>> > flinkMaster: [auto] > >>> > parallelism: 1 > >>> > runner: class org.apache.beam.runners.flink.FlinkPipelineRunner > >>> > stableUniqueNames: WARNING > >>> > streaming: false > >>> > > >>> > I noticed I was in batch mode, is there something missing in the > >>> > options > >>> > configuration, or is it that the UnsoppertedOperation is a WIP or not > >>> > supported at all by Flink ? > >>> > > >>> > And extra question, this option to define if it is in batch or stream > >>> > mode > >>> > is going to stay for long, can't be this inferred some how ? > >>> > > >>> > Thanks, > >>> > -Ismaël > >>> > > > > > >
