Re: Beam Python streaming pipeline on Flink Runner

And, as Max says, this is an SDF that wraps a BoundedSource or UnboundedSource respectively. The other way around is not possible, as SDF is strictly more powerful. On Thu, Jan 31, 2019 at 3:52 PM Robert Bradshaw wrote: > > On Thu, Jan 31, 2019 at 3:17 PM Ismaël Mejía wrote: > > > > > Not necess

Re: Beam Python streaming pipeline on Flink Runner

On Thu, Jan 31, 2019 at 3:17 PM Ismaël Mejía wrote: > > > Not necessarily. This would be one way. Another way is build an SDF wrapper > > for UnboundedSource. Probably the easier path for migration. > > That would be fantastic, I have heard about such wrapper multiple > times but so far there is

Re: Beam Python streaming pipeline on Flink Runner

I have a hard time to imagine how can we map in a generic way RestrictionTrackers into the existing Bounded/UnboundedSource, so I would love to hear more about the details. Isn't it the other way around? The SDF is a generalization of UnboundedSource. So we would wrap UnboundedSource using SD

Re: Beam Python streaming pipeline on Flink Runner

> Not necessarily. This would be one way. Another way is build an SDF wrapper > for UnboundedSource. Probably the easier path for migration. That would be fantastic, I have heard about such wrapper multiple times but so far there is not any realistic proposal. I have a hard time to imagine how ca

Re: Beam Python streaming pipeline on Flink Runner

> In addition to have support in the runners, this will require a > rewrite of PubsubIO to use the new SDF API. Not necessarily. This would be one way. Another way is build an SDF wrapper for UnboundedSource. Probably the easier path for migration. On 31.01.19 14:03, Ismaël Mejía wrote: Fortu

Re: Beam Python streaming pipeline on Flink Runner

> Fortunately, there is already a pending PR for cross-language pipelines which > will allow us to use Java IO like PubSub in Python jobs. In addition to have support in the runners, this will require a rewrite of PubsubIO to use the new SDF API. On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michel

Re: Beam Python streaming pipeline on Flink Runner

Hi Matthias, This is already reflected in the compatibility matrix, if you look under SDF. There is no UnboundedSource interface for portable pipelines. That's a legacy abstraction that will be replaced with SDF. Fortunately, there is already a pending PR for cross-language pipelines which w

Re: Beam Python streaming pipeline on Flink Runner

Hey Ankur, Thanks for the swift reply. Should I change this in the capability matrix then? Many thanks. Best, Matthias On Thu, 31 Jan 2019 at 09:31, Ankur Goenka wrote: > Hi Matthias, > > Unfortunately, unbounded reads including pubs

Re: Beam Python streaming pipeline on Flink Runner

Hi Matthias, Unfortunately, unbounded reads including pubsub are not yet supported for portable runners. Thanks, Ankur On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens wrote: > Hi everyone, > > Last few days I have been trying to run a streaming pipeline (code on > Github

Beam Python streaming pipeline on Flink Runner

Hi everyone, Last few days I have been trying to run a streaming pipeline (code on Github ) on a Flink Runner. I am running a Flink cluster locally (v1.5.6 ) I have built the SDK Harness Container: *./gradlew :beam-