Luke is right - unbounded sources should go through SDF. I am currently working on adding such support to Fn API. The relevant document is s.apache.org/beam-breaking-fusion (note: it focuses on a much more general case, but also considers in detail the specific case of running unbounded sources on Fn API), and the first related PR is https://github.com/apache/beam/pull/4743 .
Ways you can help speed up this effort: - Make necessary changes to Apex runner per se to support regular SDFs in streaming (without portability). They will likely largely carry over to portable world. I recall that the Apex runner had some level of support of SDFs, but didn't pass the ValidatesRunner tests yet. - (general to Beam, not Apex-related per se) Implement the translation of Read.from(UnboundedSource) via impulse, which will require implementing an SDF that reads from a given UnboundedSource (taking the UnboundedSource as an element). This should be fairly straightforward and will allow all portable runners to take advantage of existing UnboundedSource's. On Fri, Mar 23, 2018 at 3:08 PM Lukasz Cwik <[email protected]> wrote: > Using impulse is a precursor for both bounded and unbounded SDF. > > This JIRA represents the work that would be to add support for unbounded > SDF using portability APIs: > https://issues.apache.org/jira/browse/BEAM-2939 > > > On Fri, Mar 23, 2018 at 11:46 AM Thomas Weise <[email protected]> wrote: > >> So for streaming, we will need the Impulse translation for bounded input, >> identical with batch, and then in addition to that support for SDF? >> >> Any pointers what's involved in adding the SDF support? Is it runner >> specific? Does the ULR cover it? >> >> >> On Fri, Mar 23, 2018 at 11:26 AM, Lukasz Cwik <[email protected]> wrote: >> >>> All "sources" in portability will use splittable DoFns for execution. >>> >>> Specifically, runners will need to be able to checkpoint unbounded >>> sources to get a minimum viable pipeline working. >>> For bounded pipelines, a DoFn can read the contents of a bounded source. >>> >>> >>> On Fri, Mar 23, 2018 at 10:52 AM Thomas Weise <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> I'm looking at the portable pipeline translation for streaming. I >>>> understand that for batch pipelines, it is sufficient to translate Impulse. >>>> >>>> What is the intended path to support unbounded sources? >>>> >>>> The goal here is to get a minimum translation working that will allow >>>> streaming wordcount execution. >>>> >>>> Thanks, >>>> Thomas >>>> >>>> >>
