I guess the question is what is the code to read from an unbounded source. I have went through the documentation and it has been unclear...Spefically identifying the port to pull from.
Simply put, had a hard time understanding how to read from an unbound source. Wyatt F. On Feb 8, 2017 6:39 AM, "Jean-Baptiste Onofré" <[email protected]> wrote: > Hi > > Direct runner is able to read from unbounded source. The next transforms > of your pipeline have to deal with unbounded pcollection. > > Regards > JB > On Feb 7, 2017, at 22:16, Wyatt Frelot <[email protected]> wrote: >> >> Good evening all, >> >> I am working on project whose goal is to show the strength of beam and >> build a "pipeline" that ingests both bounded and unbounded data. >> >> I use NiFi to both "batch up" and stream the same dataset. I am running >> this using the Direct Runner >> >> Bounded: I can setup the pipeline to read and process the files. Works as >> suspected >> >> Unbounded: *Doesn't work*, but I am certain it's because I don't >> understand how the pipeline needs to be created to create the PCollection. >> I can't find any good examples >> >> #*I KNOW THIS IS WRONG* >> >> >> >> >> >> >> *StreamingOptions streamingOptions =PipelineOptions.class.;Pipeline stream = >> Pipeline.create(streamingOptions);stream.apply("WordStream", Read.from(new >> UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter<>())) >> .apply("ExtraLinesStream",ParDo.of(new AnomalyAlertDetector())) >> .apply("WriteToAccumuloStream", ParDo.of(new WriteToAccumuloStream()));* >> >> >> >>
