Given the increasing importance of multi language pipelines, it does seem that we should expand the capabilities of the DirectRunner or just go all in on FlinkRunner for testing and local / small scale development
On Wed, Dec 28, 2022 at 12:47 AM Robert Burke <rob...@frantil.com> wrote: > Probably either on Flink, or the Python Portable runner at this juncture. > > On Tue, Dec 27, 2022, 8:40 PM Byron Ellis via dev <dev@beam.apache.org> > wrote: > >> Hi all, >> >> I spent some more time adding things to my dbt-for-Beam clone ( >> https://github.com/apache/beam/pull/24670) and actually made a fair >> amount of progress, including starting to add in the profile support so I >> can start to run it against real workloads (though at the moment only the >> "test" connector is properly configured). More interestingly, though, is >> adding in support for Python Dataframe external transforms... which expands >> properly, but then (unsurprisingly) hangs if you try to actually run the >> pipeline with Java's TestPipeline. >> >> I was wondering how people go about testing Java/Python hybrid pipelines >> locally? The Java<->Python tests don't seem to actually execute a pipeline, >> but I was hoping that maybe the direct runner could be set up properly to >> do that? >> >> Best, >> B >> >
