> Fortunately, there is already a pending PR for cross-language pipelines which > will allow us to use Java IO like PubSub in Python jobs.
In addition to have support in the runners, this will require a rewrite of PubsubIO to use the new SDF API. On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <m...@apache.org> wrote: > > Hi Matthias, > > This is already reflected in the compatibility matrix, if you look under SDF. > There is no UnboundedSource interface for portable pipelines. That's a legacy > abstraction that will be replaced with SDF. > > Fortunately, there is already a pending PR for cross-language pipelines which > will allow us to use Java IO like PubSub in Python jobs. > > Thanks, > Max > > On 31.01.19 12:06, Matthias Baetens wrote: > > Hey Ankur, > > > > Thanks for the swift reply. Should I change this in the capability matrix > > <https://s.apache.org/apache-beam-portability-support-table> then? > > > > Many thanks. > > Best, > > Matthias > > > > On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goe...@google.com > > <mailto:goe...@google.com>> wrote: > > > > Hi Matthias, > > > > Unfortunately, unbounded reads including pubsub are not yet supported > > for > > portable runners. > > > > Thanks, > > Ankur > > > > On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens > > <baetensmatth...@gmail.com > > <mailto:baetensmatth...@gmail.com>> wrote: > > > > Hi everyone, > > > > Last few days I have been trying to run a streaming pipeline (code > > on > > Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner. > > > > I am running a Flink cluster locally (v1.5.6 > > <https://flink.apache.org/downloads.html>) > > I have built the SDK Harness Container: /./gradlew > > :beam-sdks-python-container:docker/ > > and started the JobServer: /./gradlew > > :beam-runners-flink_2.11-job-server:runShadow > > -PflinkMasterUrl=localhost:8081./ > > > > I run my pipeline with: /env/bin/python streaming_pipeline.py > > --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx > > --input_subscription xxx --output_subscription xxx/ > > / > > / > > All this is running inside a Ubuntu (Bionic) in a Virtualbox. > > > > The job submits fine, but unfortunately fails after a few seconds > > with > > the error attached. > > > > Anything I am missing or doing wrong? > > > > Many thanks. > > Best, > > Matthias > > > >
