> Not necessarily. This would be one way. Another way is build an SDF wrapper > for UnboundedSource. Probably the easier path for migration.
That would be fantastic, I have heard about such wrapper multiple times but so far there is not any realistic proposal. I have a hard time to imagine how can we map in a generic way RestrictionTrackers into the existing Bounded/UnboundedSource, so I would love to hear more about the details. On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <[email protected]> wrote: > > > In addition to have support in the runners, this will require a > > rewrite of PubsubIO to use the new SDF API. > > Not necessarily. This would be one way. Another way is build an SDF wrapper > for > UnboundedSource. Probably the easier path for migration. > > On 31.01.19 14:03, Ismaël Mejía wrote: > >> Fortunately, there is already a pending PR for cross-language pipelines > >> which > >> will allow us to use Java IO like PubSub in Python jobs. > > > > In addition to have support in the runners, this will require a > > rewrite of PubsubIO to use the new SDF API. > > > > On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <[email protected]> wrote: > >> > >> Hi Matthias, > >> > >> This is already reflected in the compatibility matrix, if you look under > >> SDF. > >> There is no UnboundedSource interface for portable pipelines. That's a > >> legacy > >> abstraction that will be replaced with SDF. > >> > >> Fortunately, there is already a pending PR for cross-language pipelines > >> which > >> will allow us to use Java IO like PubSub in Python jobs. > >> > >> Thanks, > >> Max > >> > >> On 31.01.19 12:06, Matthias Baetens wrote: > >>> Hey Ankur, > >>> > >>> Thanks for the swift reply. Should I change this in the capability matrix > >>> <https://s.apache.org/apache-beam-portability-support-table> then? > >>> > >>> Many thanks. > >>> Best, > >>> Matthias > >>> > >>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <[email protected] > >>> <mailto:[email protected]>> wrote: > >>> > >>> Hi Matthias, > >>> > >>> Unfortunately, unbounded reads including pubsub are not yet > >>> supported for > >>> portable runners. > >>> > >>> Thanks, > >>> Ankur > >>> > >>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens > >>> <[email protected] > >>> <mailto:[email protected]>> wrote: > >>> > >>> Hi everyone, > >>> > >>> Last few days I have been trying to run a streaming pipeline > >>> (code on > >>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink > >>> Runner. > >>> > >>> I am running a Flink cluster locally (v1.5.6 > >>> <https://flink.apache.org/downloads.html>) > >>> I have built the SDK Harness Container: /./gradlew > >>> :beam-sdks-python-container:docker/ > >>> and started the JobServer: /./gradlew > >>> :beam-runners-flink_2.11-job-server:runShadow > >>> -PflinkMasterUrl=localhost:8081./ > >>> > >>> I run my pipeline with: /env/bin/python streaming_pipeline.py > >>> --runner=PortableRunner --job_endpoint=localhost:8099 --output > >>> xxx > >>> --input_subscription xxx --output_subscription xxx/ > >>> / > >>> / > >>> All this is running inside a Ubuntu (Bionic) in a Virtualbox. > >>> > >>> The job submits fine, but unfortunately fails after a few > >>> seconds with > >>> the error attached. > >>> > >>> Anything I am missing or doing wrong? > >>> > >>> Many thanks. > >>> Best, > >>> Matthias > >>> > >>>
